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Preface 


This  study  investigated  the  effects  of  communication  delays  imposed  by  the  Voice  Switching  and 
Control  System  (VSCS)  and  satellite  linked  air  traffic  control  communications.  The  delays  were 
implemented  in  a  simulation  intercom  at  the  Federal  Aviation  Administration  Technical  Center. 
Simulation  scenarios  were  constructed  from  records  of  live  air  traffic  at  Atlanta  Air  Route  Traffic 
Control  Center.  Full  performance  level  air  traffic  control  specialists  from  the  sectors  simulated 
participated  in  the  simulation.  The  study  focused  on  predicting  whether  increased  stress,  decre¬ 
ments  in  system  performance  and  increased  communication  disruption  would  occur  due  to  the 
delays  characteristic  of  VSCS  equipment  and/or  satellite  systems,  as  compared  to  the  systems 
used  today. 

The  authors  wish  to  recognize  the  contributions  of  the  numerous  individuals  who  made  this  study 
possible.  We  are  particularly  grateful  to  the  supervisory  controllers  from  Atlanta  Center  who 
worked  extensively  with  us  at  many  points  in  the  study.  Much  of  its  success  can  be  attributed  to 
their  assistance.  We  are  also  very  grateful  to  the  ATC  specialists  from  Atlanta  Center  who  served 
as  controllers  at  the  simulation  and  to  Paul  Brinegar  from  Washington  Center  and  Dan  Johnson 
from  Chicago  Center,  who  worked  at  the  aircraft  termination  sector  for  part  of  the  study.  Many 
other  individuals  at  Atlanta  Center  deserve  our  appreciation,  especially  Bob  Owen,  who 
scheduled  controllers  for  their  participation,  and  Gary  Crosby,  for  sector  descriptions.  We  also 
greatly  appreciate  the  support  of  the  Atlanta  Center  air  traffic  facility  manager,  Stan  Ensley. 

We  also  wish  to  acknowledge  the  assistance  of  many  individuals  at  the  FAA  Air  Traffic  Plans  and 
Requirements  Service  and  Air  Traffic  Operations  Service.  Chuck  Ullman  was  instrumental  in 
making  Atlanta  Center  personnel  available  for  the  study.  Chuck  Harrison  and  Chuck  Ullman 
reviewed  and  made  useful  suggestions  regarding  the  questionnaires,  and  participated  in  the 
process  of  readying  the  simulation  scenarios.  Roy  Faber,  Chuck  Harrison  and  Robbie  McGrath 
staffed  the  aircraft  termination  sector  while  observing  the  simulation.  A1  Henry  and  Dan  Kerr 
provided  expert  advice  from  an  operational  perspective.  Ron  Morgan,  acting  manager,  and  Mitch 
Grossberg,  both  of  the  Advanced  Systems  and  Facilities  Division  at  ATR,  deserve  special  credit 
for  their  involvement  in  the  planning  of  the  project  and  thorough  review  of  an  early  draft  of  this 
report. 

Michael  Lam,  who  chairs  the  Satellite  Communications  Working  Group,  and  Glenn  Waugaman, 
manager  of  the  Telecommunications  Management  and  Operations  Division,  provided  the  neces¬ 
sary  initial  coordination  and  direction  for  the  study.  Mike  Gariazzo,  Michael  Lam,  Douglas  Lee 
and  Glenn  Waugaman  made  valuable  comments  on  an  early  draft  of  this  report. 

The  study  benefited  greatly  from  the  contributions  of  many1,  resourceful  people  at  the  FAA 
Technical  Center  including  John  Aschenbach,  Jack  Bernstein,  Ginger  Carnes,  Debbie  Cook, 
Elliot  Linsky,  Dave  Senn,  Albert  Schwartz,  Scott  Harris,  Steve  Stratoti  and  Dan  Warburten.  We 
appreciate  the  support  of  Howard  Mason,  manager  of  the  Technical  Facilities  Division,  Rene 
Matos,  supervisor  of  Simulation  Operations  and  Hugh  Milligan,  manager  of  ATC  Facilities 
Operation,  and  his  staff.  The  simulation  scenarios  were  constructed  by  SRSA  employees  Bill 
Bamberg,  Jim  Miller  and  Kevin  Walker.  CRM  employees  Dick  Algeo,  Scott  Cramer,  Scott 
Doucett,  Gwen  Harris  and  Mary  Schweiker  ran  the  simulation  and  reduced  the  data.  We  also 


wish  to  express  our  appreciation  for  the  patience  and  dedication  of  the  30  simulation  operators 
who  served  in  the  role  of  pilots,  including  lead  pilots  Dee  Algeo,  Molly  Amado  and  Mary  Rozier. 
We  also  thank  their  supervisor,  Joyce  Landing,  and  SIR  employee  Henry  Smallacomb,  and  the 
supervisor  of  the  Pilot’s  Lab,  George  Kupp,  Jr. 

The  authors  wish  to  express  their  appreciation  to  a  number  of  individuals  at  RSPA/TSC:  Dr.  Kim 
Cardosi,  Operator  Performance  and  Safety  Analysis  Division,  for  insightful  contributions  toward 
planning  the  study,  and  for  her  work  in  planning  and  executing  data  collection,  and  Bill  Hill,  Chief 
Training  Officer,  Human  Resources  Division,  for  innovative  contributions  to  the  data  collection 
procedures.  Neil  Patt,  Chief,  and  Ben  Goldstein,  both  of  the  Telecommunications  Division,  made 
many  valuable  comments  on  an  early  draft  of  the  report.  We  also  are  grateful  to  Pamela  Boole  of 
RSPA/TSC,  Jonathan  Belcher  of  Superior  Design,  and  Margaret  Warner  of  EG&G  Dynatrend 
for  their  careful  recording  and  verification  of  the  data,  and  Capt.  John  Turner  of  EG&G 
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the  quality  of  the  simulation.  Professor  Phil  Sampson  of  Tufts  University  reviewed  and  con¬ 
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This  study  was  sponsored  by  the  Federal  Aviation  Administration,  System  Maintenance  Service 
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Executive  Summary 


Introduction 

The  purpose  of  this  study  is  to  investigate  the  effects  of  the  delays  imposed  by  the  use  of 
geosynchronous  satellites  and  the  Voice  Switching  and  Control  System  (VSCS)  on  air  traffic 
controller/pilot  communications,  stress,  and  system  performance.  FAA  satellite  based  systems 
now  operate  successfully  in  Alaska  and  the  Caribbean  under  moderate  traffic  conditions.  Con¬ 
versations  with  supervisory  staff  at  Transport  Canada  indicate  that  Canadian  ATC  systems  also 
use  satellite  communications  successfully.  No  operational  experience  is  available  for  VSCS 
systems.  This  study  provides  information  on  the  effects  of  communications  delay  under  a  wide 
variety  of  conditions. 

The  study  examines  the  effects  of  satellite  and  VSCS  communication  delays  on  simulated  air 
traffic  control  system  performance.  Four  delays  were  implemented  in  the  simulation,  those 
imposed  by  the  present  system  alone  (Today),  VSCS  alone  (VSCS),  Today  with  Satellite 
(Today  +  Sat),  and  VSCS  with  Satellite  (VSCS  +  Sat). 

The  nine-day  simulation  was  conducted  at  the  FAA  Technical  Center.  Simulation  scenarios  were 
constructed  from  recordings  of  actual  air  traffic  at  Atlanta  Air  Route  Traffic  Control  Center 
(ARTCC).  Five  adjacent  sectors  of  the  ARTCC  were  simulated.  Three  sectors  were  designated 
test  sectors  and  the  two  others  were  considered  non-test  sectors.  The  test  sectors  included  one 
low  altitude  sector  (Sinca  Low)  and  two  high  altitude  sectors  (Dublin  High  and  Macon  High).  Tiie 
study  focused  only  on  events  related  to  the  test  sectors.  The  non-test  sectors  were  used  to  support 
the  test  sectors  by  realistically  receiving  and  transferring  aircraft.  The  subjects  were  nine  full 
performance  level  (FPL)  air  traffic  control  specialists  currently  working  in  the  Atlanta  Center 
area  containing  the  sectors  simulated,  assisted  at  the  data  controller  positions  by  an  equal  number 
of  FPL  controllers  from  the  same  center.  Thirty  simulation  operators,  who  had  been  trained  by 
the  FAA  Technical  Center,  performed  in  the  role  of  pilots. 

Three  levels  of  communications  workload  were  established  by  creating  three  scenarios  with 
different  numbers  of  aircraft.  These  air  traffic  load  numbers  were  based  upon  Atlanta  Center 
traffic  load  norms  for  the  actual  sectors.  Each  of  the  28  test  sessions  included  one  of  the  four  types 
of  delay  and  one  of  the  three  levels  of  communications  workload. 


The  following  areas  were  examined  for  potential  effects  of  delay: 

•  System  performance 

•  Communications  disruptions 

•  Controller  stress,  effort  and  attention  requirements 
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System  performance  measures  included 

-  Separation  infringements 

-  Numbers  of  aircraft  handled 

Communications  disruptions  consist  of  “step-ons”  and  verbal  mistakes. 

-  Step-ons  comprise  Pilot-Controller/Controller-Pilot  step-ons  (the  pilot 
blocks  the  transmission  by  the  controller  or  the  controller  blocks  the  trans¬ 
mission  by  the  pilot),  and  Pilot-Pilot  step-ons. 

-  The  verbal  mistakes  recorded  consist  of  wrong  information  that  was  cor¬ 
rected  in  the  same  transmission,  and  uncorrected  wrong  information. 

Estimates  of  controller  stress  and  attention  were  assessed  through  questionnaires  filled  out  by  the 
controller  subjects  and  by  operations  observers  (supervisory  FPL  controllers  from  the  Atlanta 
ARTCC). 

Of  these,  communications  disruptions  and  especially  step-ons  (call  blocking)  were  considered  to 
be  the  most  sensitive  measure  of  the  effect  of  delay. 

This  study  focused  on  predicting  whether  the  delays  characteristic  of  VSCS  equipment  and/or 
satellite  systems,  as  compared  to  the  systems  used  today,  would  cause  deterioration  in  system 
performance,  increased  controller  stress,  and  increased  communications  disruptions.  Three 
hypotheses  were  tested  in  this  study: 

•  System  performance  is  degraded  with  increased  delay. 

•  Controller  stress  increases  with  delay. 

•  Communications  disruptions  increase  with  delay. 

Results 

System  Performance 

•  No  statistically  significant  differences  were  found  between  any  of  the  delay  con¬ 
ditions,  satellite  or  VSCS,  in  measures  of  separation  infringements  or  numbers 
of  aircraft  handled. 

Controller  Stress 

•  No  statistically  significant  differences  were  found  between  any  of  the  delay  con¬ 
ditions  in  measures  of  stress,  effort,  or  attention. 
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Communications  Disruptions 

•  No  statistically  significant  increases  in  communications  disruptions  were  found 
which  could  be  attributed  to  simulated  VSCS  delay  conditions. 

•  No  statistically  significant  differences  in  verbal  mistakes  were  found  between 
any  of  the  delay  conditions. 

•  Statistically  significantly  more  Pilot-Controller/Controller-Pilot  step-ons  were 
recorded  for  satellite  conditions  than  non-satellite  conditions  at  the  highest 
workload  levels  tested. 

Conclusions  and  Recommendations 

Increased  step-ons  were  found  at  the  highest  workload  levels  under  satellite  delay  conditions.  The 
question  remains:  Will  the  observed  increase  in  step-ons,  under  satellite  delay  conditions  at  high 
communication  workloads  translate  into  inferior  real-world  system  performance?  This  matter  can 
best  be  resolved  by  making  a  separate  field  study  using  satellite  communications  within  sectors 
with  high  communications  activity. 


xi/xii 


1.  Introduction 


The  purpose  of  this  study  is  to  investigate  the  effects  of  the  delays  imposed  by  the  use  of 
geosynchronous  satellites  and  the  Voice  Switching  and  Control  System  (VSCS)  on  air  traffic 
controller  communications,  workload,  and  system  performance.  The  study  was  conducted  with 
full  performance  level  (FPL)  controllers  using  the  real  time  air  traffic  control  simulation  facilities 
at  the  Federal  Aviation  Administration  Technical  Center. 1 

This  study  focuses  on  predicting  whether  increased  communications  disruptions,  increased  con¬ 
troller  stress,  and  degradation  in  system  performance  will  occur  with  the  delays  characteristic  of 
VSCS  equipment  and/or  satellite  systems,  as  compared  to  systems  used  today.  The  tests  were 
conducted  under  conditions  that  realistically  simulated  pilot  contention,  with  air  traffic  loads  that 
would  produce  a  range  of  communications  loads.  Five  adjacent  en  route  sectors  south  of  Atlanta 
were  chosen  for  the  simulation.  Performance  was  studied  in  three  “test  sectors”:  simulations  of 
Dublin  High,  Macon  High  and  Sinca  Low.  Two  other  sectors  (simulations  of  Clark  Hill  Ultra 
High  and  South  Departure  Low)  supported  the  test  sectors  in  the  simulation.  The  simulations 
consisted  of  scenarios  constructed  from  recordings  of  air  traffic  in  the  corresponding  actual 
sectors. 

The  subjects  were  FPL  controllers  who  normally  work  in  the  Atlanta  Center  area  containing  the 
sectors  corresponding  to  the  simulated  sectors.  Their  performance  was  observed  by  supervisors 
from  Atlanta  Center.  Their  communications  were  observed  and  taped  by  personnel  from  the 
Transportation  Systems  Center.  The  supervisors  and  subjects  filled  out  questionnaires  following 
each  simulation  session.  In  addition,  a  variety  of  data  was  recorded  automatically  by  the  simula¬ 
tion  computer  systems.  It  was  anticipated  that  these  efforts  would  provide  a  valid  and  reliable 
statistical  basis  for  decisions  which  pertain  to  the  use  of  the  VSCS  and  satellite  communications 
systems  in  future  ATC  communications. 


1  The  study  was  previously  described  in  the  document  A  TC  Voice  Communications  Delay  Test  Plan  (G.  Spanier, 
17  October  1989),  circulated  by  the  Federal  Aviation  Administration  Technical  Center.  (See  Appendix  A) 


The  communications  systems  that  were  simulated  impose  three  types  of  delay: 

•  A  “ground-to-air  set-up  delay”  occurring  between  the  time  the  controller’s 
microphone  is  keyed  and  the  time  that  a  message  can  be  accepted  by  the  ground 
transmission  system.  The  message  is  “clipped”  (i.e.,  the  initial  segment  is  lost)  if 
it  begins  before  the  set-up  delay  elapses. 

•  A  “ground-to-air  propagation  delay”  occurring  between  the  time  the  message 
enters  the  system  and  the  time  the  message  is  received.  The  message  is  delayed 
until  the  transmission  is  propagated  through  the  system. 

•  An  “air-to-groun  i  propagation  delay”  occurring  between  the  time  the  pilot 
makes  a  transmission  and  the  time  it  is  received  at  the  en  route  center.  The  mes¬ 
sage  is  delayed  until  the  transmission  is  propagated  through  the  system. 

Four  combinations  of  set-up  and  propagation  delays  were  studied,  corresponding  to  the  delay 
values  associated  with: 

1.  currently  used  systems  (Today) 

2.  future  systems  incorporating  elements  of  currently  used  systems  and  Voice  Switch¬ 
ing  and  Control  System  equipment  (VSCS) 

3.  currently  used  systems  incorporating  satellite  links  (Today  +  Sat.) 

4.  future  systems  incorporating  elements  of  currently  used  systems  and  VSCS  equip¬ 
ment  with  satellite  links  (VSCS  +  Sat) 

The  delay  values  that  were  used  were  developed  by  the  FAA  Technical  Center  (See  Appendix  B 
for  a  description  of  the  considerations  used  to  determine  these).  The  delays  range  from  those 
corresponding  to  current  equipment  (225  msec  ground-to-air  set-up  delay  and  no  air-to-ground 
delay)  to  satellite  delays  that  include  260  msec  propagation  delays  in  both  the  air-to-ground  and 
ground-to-air  directions  plus  set-up  delays.These  delays  were  implemented  in  an  intercom  system 
at  the  Federal  Aviation  Administration  Technical  Center  simulation  facility. 

The  following  three  hypotheses  were  tested  in  this  study: 

•  System  Performance  is  degraded  with  increased  delay. 

•  Controller  stress  increases  with  delay. 

•  Communications  disruptions  increase  with  delay. 


These  hypotheses  cannot  be  tested  with  equal  sensitivity  in  this  study.  The  test  of  system 
performance  degradation  depends  to  a  large  extent  on  our  ability  to  detect  changes  in  the 
frequency  of  separation  infringements.  These  infringements  occur  very  infrequently  in  the  real 
world  (For  Atlanta  historically,  less  than  one  for  every  30,000  aircraft  handled  2).  Our  study 
provides  fewer  than  5,000  aircraft  handled.  Therefore  it  would  take  a  very  major  change  in  the 
frequency  of  infringements  due  to  communications  delay  to  be  statistically  significant. 

Changes  in  stress  are  readily  measurable  but  in  this  simulation  the  level  of  stress  and  workload  is 
due  directly  to  the  number  of  aircraft  handled  and  is  only  an  indirect  side-effect  of  communica¬ 
tions  disruptions,  particularly  step-ons. 

In  this  study  the  frequency  of  step-ons  is  by  far  the  most  sensitive  measure  of  the  impact  of 
communications  delay.  Step-ons  are  a  direct  result  of  delay,  occur  with  sufficient  frequency  in  the 
real-world  to  be  statistically  testable,  and  are  probably  the  source  of  the  other  problems  we  are 
testing  for. 


2  FAA  Office  of  Aviation  Safety.  Profile  of  Operational  Errors  in  the  National  Airspace  System,  1987 
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2.  Method 

2.1  Overview 

The  study  was  conducted  with  current,  qualified,  full  performance  level  (FPL)  controllers  using 
the  real  time  air  traffic  control  simulation  facilities  at  the  FAA  Technical  Center.  Nine  FPL 
controllers  from  the  area  of  specialization  corresponding  to  the  test  airspace  participated  in  the 
radar-controller  (R-controller)  positions  as  subjects.  They  were  responsible  for  all  communica¬ 
tions  with  pilots.  An  equal  number  of  FPL  controllers  from  Atlanta  Center  staffed  the  data 
positions  (D-controllers).  They  handled  all  ground-to-ground  communications. 

Supervisors  from  other  areas  of  specialization  at  Atlanta  Center  observed  the  performance  of  the 
R-controllers.  Thirty  simulation  operators,  trained  to  serve  in  the  role  of  simulation  pilots,  also 
participated.  All  communications  were  observed  by  trained  personnel  from  the  Transportation 
Systems  Center.  Data  relating  to  controller  stress,  system  performance,  and  communications 
disruption  were  recorded  by  the  observers,  the  R-controllers,  and  the  simulation  computer 
system. 

There  were  28  test  sessions  during  the  three-week  study.  A  different  controller  team  participated 
each  week  for  three  days.  There  were  three  sessions  each  day,  except  for  one  day  when  there  were 
four.  The  fourth  session  was  added  because  of  equipment  problems  during  the  first  session  of  the 
first  day. 

2.2  Background 

Communication  delays  have  the  potential  for  causing  problems  in  air  traffic  control  (ATC) 
communications.  One  type  of  communications  disruption  is  the  “step-on.”  In  a  step-on,  one  partv 
blocks  a  communication  from  another  party  by  starting  a  communication  after  the  other  star*, 
one,  but  before  it  is  completed. 

Pilot-controller  step-ons  occur  if  a  pilot  blocks  a  call  from  the  controller.  Communications  delays 
can  cause  this  type  of  step-on  if  a  pilot  initiates  a  transmission  between  the  time  the  controller 
keys  the  microphone  to  transmit  information  to  that  pilot,  and  the  time  the  controller’s  message 
arrives.  Because  neither  one  can  receive  a  message  while  keying  the  microphone,  all  or  part  of  the 
incoming  message  may  be  lost. 

Delay  can  also  cause  controller-pilot  step-ons  if  the  pilot  initiates  a  transmission  but  the  control¬ 
ler  is  unaware  of  it  on  account  of  equipment  and  propagation  delays  and  initiates  a  transmission 
before  the  pilot’s  transmission  can  be  received.  Here  the  controller  blocks  the  transmission  by  the 
pilot. 

Pilot-pilot  step-ons  are  not  expected  to  be  affected  by  delay  because  no  delay  was  imposed  on  the 
intercom  channels  that  connected  pilots  in  the  same  sector.  This  arrangement  accurately  simu¬ 
lated  actual  operations.  Pilot-pilot  step-ons  (as  well  as  some  controller-pilot  and  pilot-controller 
step-on)  probably  result  from  message  overlap  during  periods  of  frequent  communication.  In  an 
operational  environment,  a  pilot-pilot  step-on  results  in  a  “squeal”  such  that  the  controller  cannot 
comprehend  the  overlapping  portions  of  the  pilots’  transmissions. 
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2.3  Test  Airspace 

The  test  airspace  involved  five  contiguous  en  route  sectors  based  upon  particular  Atlanta  AKTCC 
sectors.  Performance  was  studied  in  three  “test  sectors.”  These  three  were  supported  by  two 
“non-test  sectors,”  which  realistically  transferred  and  received  aircraft.  Performance  in  the 
non-test  sectors  was  not  studied.  The  test  sectors  were  developed  from  recordings  of  live  air  traffic 
in  corresponding  sectors  described  by  Atlanta  Center  as  follows: 1 

Dublin  High:  “Air  carrier  aircraft  generate  the  majority  of  traffic  handled  by  this  sector.  This 
sector  provides  IFR  arrival  service  to  the  Atlanta  Terminal  area  and,  at  the  same  time,  handles  a 
large  amount  of  en  route  traffic.  A  moderate  amount  of  traffic  is  generated  by  aircraft  transition¬ 
ing  to/from  Macon,  GA/Robins  AFB  and  adjacent  airports.  Traffic  flow  is  predominantly 
northwest/southeast  with  numerous  departures  from  Atlanta  and  arrivals  to  the  East  Coast 
interspersed  with  Atlanta  area  arrivals.  The  high  complexity  of  this  sector  is  created  by  Atlanta 
terminal  area  arrivals  entering  the  sector  at  several  locations  that  require  spacing  while  continu¬ 
ing  to  provide  service  to  en  route  traffic.  A  large  amount  of  coordination  is  required  with  the 
Sinca  sector  in  order  to  achieve  the  required  intrail  spacing.  The  unique  characteristic  of  this 
sector,  which  increases  the  complexity,  is  the  requirement  for  the  controllers  to  change  altitudes 
of  a  large  percentage  of  aircraft  to  conform  with  letters  of  agreement  and  traffic  flow.  These 
situations  add  to  the  sector’s  complexity  and  necessitate  careful  planning  and  coordination.  This 
sector  daily  works  F-15  aircraft  to/from  Robins  AFB." 

Macon  High:  “This  sector  provides  IFR  service  from  the  Atlanta  terminal  area  with  a  mixture  of 
a  proportionate  amount  of  en  route  traffic.  Traffic  flow  is  predominantly  north/south  with  a 
moderate  amount  of  crossing  traffic.  Military  operations  generate  additional  traffic  which  must 
be  blended  with  normal  traffic.  The  unique  characteristic  of  this  sector  is  the  fact  that  controllers 
are  required  to  change  altitudes  on  all  J45  traffic.  Obviously  this  built-in  head-on  situation 
increases  the  sector’s  complexity  and  necessitates  careful  planning  and  coordination.” 

Sinca  Low:  “This  sector  provides  IFR  arrival  service  to  the  Atlanta  terminal  area  and,  at  the  same 
time,  handles  a  proportionate  amount  of  en  route  traffic.  Traffic  flow  is  predominantly 
northwest/southeast.  Military  operations  generate  a  moderate  amount  of  traffic  which  must  be 
blended  with  the  traffic  flow.  Controllers  are  required  to  provide  arrival  spacing  for  the  Atlanta 
terminal  area  which  requires  careful  planning  and  a  large  amount  of  coordination  with  the  Dublin 
High  altitude  sector." 

The  two  non-test  sectors  correspond  to  Clark  Hill  Ultra-High  and  South  Departure  Low.  In 
addition,  an  aircraft  termination  sector  was  included  in  the  simulation  to  represent  a  termination 
point  for  the  aircraft.  Aircraft  disappeared  from  simulated  radar  following  transfer  to  this  sector. 
Airspace  diagrams  for  the  actual  sectors  are  reproduced  in  Figure  1  (Dublin  High),  Figure  2 
(Macon  High),  and  Figure  3  (Sinca).  An  extended  low  altitude  view  of  the  sectors  simulated  is 
reproduced  in  Figure  4.  Ratings  of  the  realism  of  the  simulated  air  traffic  are  provided  in 
Appendix  C. 


1  Quoted  sector  descriptions  were  obtained  from  Atlanta  ARTCC,  1989.  Dublin  High  and  Sinca  Low  have 
since  been  changed  at  Atlanta  Center. 
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2.4  independent  Variables 

2.4.1  Delay 

The  following  values  were  obtained  from  the  FAA  Technical  Center.  The  Today  delay  condition 
simulated  current  equipment  delays  by  imposing  on  the  simulation  intercom  circuitry  a  225  msec 
ground-to-air  set-up  delay,  no  ground-to-air  propagation  delay,  and  no  air-to-ground  propagation 
delay.  The  VSCS  condition  simulated  delays  characteristic  of  VSCS  and  associated  equipment  by 
imposing  a  99  msec  ground-to-air  set-up  delay,  a  70  msec  ground-to-air  propagation  delay,  and  a 
70  msec  air-to-ground  propagation  delay.  Today  +  Sat  provided  a  225  msec  ground-to-air  set-up 
delay,  a  260  msec  ground-to-air  propagation  delay,  and  a  260  msec  air-to-ground  propagation 
delay.  VSCS  +  Sat  imposed  a  99  msec  ground-to-air  set-up  delay,  a  330  msec  ground-to-air 
propagation  delay,  and  a  330  msec  air-to-ground  propagation  delay. 

The  delay  values  are  shown  in  Figure  5.  The  total  height  of  the  stacked  bars  shows  the  interval 
during  which  a  pilot  could  unknowingly  block  the  message  of  a  controller  who  is  calling  immedi¬ 
ately  after  another  pilot’s  call.  Only  air-to-ground  and  ground-to-air  delays  were  simulated; 
ground-to-ground  communications  delays  may  be  characteristic  of  VSCS  and  satellite  links,  but 
were  not  imposed  due  to  limitations  of  the  simulation  facility.  In  the  simulation,  pilots  whose 
microphones  were  unkeyed  could  hear  other  pilots  in  their  sector  with  no  delay. 
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Figure  2.  Airspace  Diagram  for  Macon  High 
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Figure  3.  Airspace  Diagram  for  Sinca  Low 
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Figure  4.  Extended  Low  Altitude  View  of  Simulated  Locations 
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2.4.2  Communications  Workload 

Communications  workload  levels  were  based  upon  traffic  load  reference  values  for  the  three  test 
sectors.  The  100%  “peak”  values  were  obtained  from  Atlanta  Center.  The  manner  in  which  these 
average  values  were  calculated  is  described  in  the  Test  Plan  (Appendix  A).  The  100%  reference 
values  for  Dublin,  Macon  and  Sinca  were  12.0,  10.0,  and  12.0,  respectively.  These  values  repre¬ 
sent  the  number  of  aircraft  in  the  sector  during  a  five  minute  interval. 

Three  levels  of  communications  workload  were  developed  for  the  study:  “Medium  load”  was 
70%,  “high  load”  was  90%,  and  “very  high  load”  was  110%.  Prior  to  the  study,  operations 
observers  (supervisory  controllers  from  Atlanta  Center)  assessed  the  realism  of  the  communica¬ 
tions  load  levels.  The  communications  loads  were  accordingly  supplemented  with  additional  pilot 
calls  (e.g.,  requests  for  direct  routing).  Thirteen  calls  were  added  for  the  high  load  condition  and 
fourteen  for  very  high  load.  Tables  1  and  2  present  the  resulting  numbers  of  aircraft  and 
communications  at  each  level  of  communication  workload  for  the  three  sectors. 

2.4.3  Sector  and  Subject 

The  sector  variable  is  intended  to  capture  sector  differences.  The  subject  variable  is  intended  to 
account  for  differences  among  the  R-controllers. 

2.5  Sequence  of  Test  Conditions 

The  sequence  of  test  conditions  is  shown  in  Table  3.  Sector  assignments,  which  are  described 
below,  are  indicated  as  letters  in  this  table.  Each  controller  team  participated  in  sessions 
corresponding  to  one  row,  and  consisted  of  three  pairs  (each  with  an  R-controller  and  a  D-con- 
troller). 

The  first  session  of  each  week’s  testing  began  with  a  low  delay  condition  (Today  or  VSCS)  to 
prevent  any  potential  disorientation  caused  by  the  high  delay  conditions  from  affecting  the  low 
delay  conditions.  Following  this  session,  the  12  combinations  of  delay  and  communications 
workload  were  presented  in  an  order  that  would  result  in  the  maximum  number  of  sessions 
between  occurrences  of  the  same  delay,  occurrences  of  the  same  communications  workload,  and 
occurrences  of  the  same  sector  assignments.  Nonetheless,  some  conditions  were  repeated  in 
neighboring  sessions. 

Three  sessions  were  presented  each  day,  except  for  the  second  day  of  the  third  week  when  a  fourth 
session  was  presented.  This  fourth  session,  which  was  not  in  the  planned  sequence  of  sessions, 
incorporated  the  VSCS  delay  at  the  very  high  level  of  communications  workload,  with  sector 
assignment  “a.”  It  is  not  shown  in  Tables  3  and  4.  This  session,  moreover,  was  added  because  of 
equipment  problems  during  the  first  session. 
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Table  1.  Mean  Traffic  Density  by  Communications  Workload  and  Sector 


Communications 

Workload 

Dublin  High 

Macon  High 

Sinca  Low 

Medium 

7.37 

7.48 

7.87 

High 

10.54 

10.49 

10.17 

Very  High 

13.7 

11.13 

13.92 

Note:  Numbers  indicate  aircraft  within  the  sector  over  a  five  minute  interval. 


Table  2.  Mean  Communications  Activity  by  Load  and  Sector 


Communications 

Load 

Dublin  High 

Macon  High 

Sinca  Low 

Medium 

Per  Session 

133.50 

97.38 

164.25 

Per  Minute 

2.02 

1.47 

2.47 

High 

Per  Session 

210.25 

153.25 

239.25 

Per  Minute 

2.98 

2.18 

3.39 

Very  High 

Per  Session 

297.33 

180.44 

275.44 

Per  Minute 

4.12 

2.50 

3.81 

Note:  Communications  Activity  as  indicated  by  controller  microphone  keypresses  (push-to-talk  activity). 
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Table  3.  Sequence  of  Test  Conditions 


Team 

Session  Number 

And 

Week 

t 

2 

3 

4 

5 

6 

7 

8 

9 

1 

OVH 

1H 

3M 

2VH 

OH 

0M 

i  VH 

HB| 

c 

a 

b 

a 

b 

a 

b 

mm 

2 

1H 

0M 

2VH 

3M 

OH 

OVH 

2H 

1  1 

c 

a 

b 

c 

b 

c 

a 

1 

mm 

3 

1M 

3VH 

2H 

1 

3H 

OVH 

2M 

1VH 

OH 

c 

a 

C 

mm 

b 

c 

a 

b 

a 

The  delay  conditions  are  Today  (0),  VSCS  (1),  Today  +  Sat  (2)  and  VSCS  +  Sat  (3).  The  communication  workload 
conditions  are  Medium  (M).  High  (H),  and  Very  High  (VH).  Sets  of  controller-sector  pairings  are  indicated  by  the 
letters  a,  b,  and  c.  Each  cell  represents  three  observations,  one  at  each  sector. 


2.6  Counterbalancing 

2.6.1  Subjects,  Delay,  and  Communications  Workload 

All  of  the  28  simulation  sessions  combined  one  of  four  delay  conditions  (Today,  VSCS, 
Today  +  Sat,  and  VSCS  +  Sat)  and  one  of  three  communications  workload  conditions  (Medium, 
High,  and  Very  High).  Every  delay  and  load  combination  was  presented  twice,  except  for  the 
three  Today  delay  conditions  (one  with  each  traffic  load),  each  of  which  was  presented  three  times, 
and  the  VSCS  delay  condition  at  the  very  high  communication  workload  level,  which  was  also 
presented  three  times.  Each  combination  was  presented  concurrently  in  all  three  test  sectors. 

Every  controller  worked  at  the  same  sector  during  three  sessions,  once  with  each  of  the  three 
levels  of  communications  workload.  Controllers  participating  during  the  third  week  also  par¬ 
ticipated  in  the  added  (fourth)  session  on  the  second  day. 

The  combinations  of  delay  and  communications  workload  presented  to  the  three  teams  of  subject 
controllers  during  their  respective  test  weeks  (one  team  each  week)  are  shown  in  Table  4.  The 
information  in  Table  4  can  be  derived  from  Table  3. 
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Table  4.  Assignment  of  Controller  Teams  to  Test  Conditions 


Team  and  Week 

Communication  Workload 

Sector 

Medium 

High 

Very  High 

Assignment 

0 

1 

2 

a 

1 

3 

0 

1 

b 

2 

3 

0 

c 

0 

2 

3 

a 

2 

1 

0 

2 

b 

3 

1 

0 

c 

2 

0 

3 

a 

3 

0 

3 

1 

b 

1 

2 

0 

c 

The  delay  conditions  are  Today  (0),  VSCS  (1),  Today  +  Sat  (2)  and  VSCS  +  Sat  (3). 

2.6.2  Subjects  and  Sector 

The  R-controlIers  were  assigned  at  random  to  their  first  sector,  and  then  rotated  from  one  sector 
to  another  according  to  a  pre-arranged  schedule.  The  D-controllers  were  paired  with  the 
R-controllers  throughout  the  week  (three  days)  during  which  they  participated,  and  rotated  with 
them.  The  controllers  who  worked  at  the  non-test  sectors  remained  at  those  sectors  for  the  entire 
week. 

The  sector  rotation  schedule  consisted  of  three  sets  of  controller-sector  pairings.  In  the  first  set 
("a"  in  Tables  3  &  4),  R-Controller  1  was  assigned  to  Sector  1,  R-Controller  2  to  Sector  2,  and 
R-Controller  3  to  Sector  3.  In  the  second  set  ("b"  in  Tables  3  &  4),  R-Controller  1  was  assigned  to 
Sector  2,  R-Controller  2  to  Sector  3,  and  R-Controller  3  to  Sector  1.  In  the  third  set  ("c"  in  Tables 
3  &  4),  R-Controller  1  was  assigned  to  Sector  3,  R-Controller  2  to  Sector  1,  and  R-Controller  3  to 
Sector  2.  The  three  sets  of  sector  assignments  are  summarized  in  Table  5. 
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Table  5.  Assignment  of  Controllers  to  Sectors 


Assignment 

R  Controller  No. 

Sector 

a 

1 

1 

2 

2 

3 

3 

b 

1 

2 

2 

3 

3 

1 

c 

1 

3 

2 

1 

3 

2 

2.7  Dependent  Variables 

Three  categories  of  dependent  variables  were  chosen: 

(1)  controller  stress 

(2)  system  performance 

(3)  communications  disruption 

The  stress  category  consists  of  the  demands  made  on  the  attention  of  the  controller,  stress  itself, 
and  overcontrol,  i.e.,  acting  on  the  problems  too  early. 

System  performance  includes  separation  infringements,  the  number  of  aircraft  handled,  and 
smoothness  of  air  traffic  flow.  Separation  infringements  are  violations  of  aircraft  proximity 
standards.  Aircraft  are  said  to  be  “handled”  following  contact  by  the  controller.  Smoothness  of 
air  traffic  flow  is  defined  by  the  extent  of  any  disruption  in  the  orderly  movement  of  aircraft. 

Communications  disruption  consists  of  step-ons,  corrected  verbal  mistakes,  and  uncorrected 
verbal  mistakes.  Step-ons  are  defined  as  overlapping  transmissions.  Corrected  and  uncorrected 
verbal  mistakes  include  such  events  as  uttering  incorrect  call  signs  or  frequencies.  Corrected 
mistakes  are  those  corrected  in  the  same  transmission;  uncorrected  mistakes  include  those 
corrected  in  a  later  transmission  and  uncorrected  mistakes.  Section  2.10.3  elucidates  the  manner 
in  which  data  relating  to  the  dependent  variables  was  collected. 

2.8  Equipment 
2.8.1  Facility 

The  test  bed  consisted  of  three  elements  of  the  FAA  Technical  Center:  the  En  Route  System 
Support  Facility  (ESSF),  the  Host  Computer  System  Support  Facility  (Host  SSF),  the  NAS 
Simulation  Support  Facility  (NSSF),  and  the  AMECOM  communications  system. 

The  Host  SSF  consisted  of  processing  and  peripheral  support  equipment  which  is  identical  to 
present  en  route  center  equipment,  especially  the  IBM  3083  central  processor  and  its  direct 
support  units  and  peripheral  devices,  their  interfaces  and  processors.  Keyboards,  displays,  ter¬ 
minals,  and  printers  were  all  identical  to  current  field  equipment.  Software  was  identical  to  that 
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used  in  real  en  route  centers  for  both  actual  operations  and  data  collection  and  analysis.  A  9020E 
computer  (IBM  360/65)  was  used  as  the  display  channel  processor  for  the  generation  of  surveil¬ 
lance  plan  view  information  and  other  alphanumeric  data  on  plan  view  (simulated  radar)  displays 
(PVDs). 

The  NSSF  Controller  Laboratory  consisted  of  processors,  interface  equipment,  and  displays  to 
perform  three  major  simulation  functions:  (1)  the  generation  of  flight  paths,  plans,  and  strips  for 
simulated  aircraft,  (2)  the  scripted  prompting  and  direction  of  simulator  pilots  to  fly  those  flight 
paths  and  interact  with  the  controllers  and  (3)  the  generation  of  the  targets  and  associated 
alphanumeric  data  to  the  Host  SSF  as  simulated  radar  data  inputs. 

The  AMECOM  communications  system  consisted  of  audio  communication  control,  distribution 
and  recording  configured  to  provide  the  operational  intercom,  interphone,  and  air-ground  inter¬ 
sector  and  intra-sector  communications  among  controllers,  and  between  controllers  and  simula¬ 
tion  pilots.  Digital  recordings  of  controller  microphone  keying  (push-to-talk  switch  actions)  and 
channel  selections,  as  well  as  analog  recordings  of  receptions  and  transmissions  by  each  control¬ 
ler,  were  provided. 

The  AMECOM  system  was  modified  to  provide  time-coordinated  and  allocated  transmitter 
turn-on  and  propagation  delays.  It  was  further  modified  to  provide  an  auditory  cue  to  controllers, 
corresponding  to  a  side-tone  change  in  received  audio  when  the  transmitter  was  enabled.  All 
delay  effects  and  values  were  independently  controlled  and  calibrated  prior  to  each  session. 

As  in  actual  ATC  communications,  when  either  the  controller  or  pilot  keyed  the  microphone, 
others  transmitting  on  the  same  “frequency”  became  inaudible.  Thus  each  of  the  pilots  assigned 
to  the  same  sector  (and  communications  channel)  could  hear  all  of  the  communications  on  that 
channel  except  when  that  pilot’s  microphone  was  keyed.  However,  unlike  the  real  world,  pilot- 
pilot  step-ons  did  not  result  in  a  distracting  squeal  and  blocking;  rather,  the  two  voices  could  both 
be  heard. 

The  simulated  sector  frequencies  (e.g.,  111.1)  were  realistic,  but  not  the  same  as  those  used  at 
Atlanta  Center.  These  “frequencies,”  which  were  used  to  realistically  transfer  aircraft  between 
sectors,  designated  the  intercom  channels  connecting  the  controller  and  pilot  positions. 1 

2.8.2  Data  Recording  Equipment 

Each  of  the  observers  used  a  Radio  Shack  100  lap-top  computer  programmed  for  recording 
step-ons  and  mistakes.  These  events  and  the  time  of  each  entry  were  recorded  by  single 
keystrokes.  A  map  light  was  attached  to  each  lap-top  so  that  its  screen  could  be  read  in  the 
realistically  dim  light  without  causing  glare  on  the  PVD.  Voice  recordings  of  all  pilot  and 
controller  voice  communications  (by  sector)  were  made  with  cassette  tape  recorders  from  ob¬ 
server  position?  in  the  pilot  laboratory.  Data  was  also  recorded  by  the  simulation  computer 
system.  Th;s  data  included  numbers  of  aircraft  handled,  controller  transmissions,  and  separation 
infringements. 


2  See  for  further  details  FAA  National  Airspace  System  En  Route  System  Support  Facility  Laboratory  Handbook. 
NASP-5204-04,  1989 


2  -  14 


2.8.3  Simulation  Scenarios 

Scenarios  representing  the  three  levels  of  communications  workload  differed  primarily  because 
of  flights  added  to  the  high  and  very  high  load  scenarios.  Scenario  development  involved  acquisi¬ 
tion  and  adaptation  of  data  on  actual  Atlanta  Center  traffic.  Scenarios  were  constructed  for 
typical  conditions  in  the  test  sectors.  Events  that  occur  less  than  once  per  month,  such  as 
emergency  lifeguard  flights  and  other  emergencies,  were  not  included.  Weather  incidents  were 
also  left  out.  Supervisory  controllers  from  Atlanta  Center  (the  operations  observers)  assisted  with 
scenario  development. 

2.9  Personnel 

Information  concerning  the  communications  workload  and  delay  conditions  was  not  revealed  to 
the  participants  or  observers.  All  were  told  that  the  tests  were  being  conducted  to  investigate  the 
effects  of  equipment  delays. 

2.9.1  Controllers 

Nine  test  sector  R-controllers  participated  as  subjects  in  the  study,  three  each  week.  They  all  had 
at  least  two  years  of  FPL  experience,  with  a  mean  of  5.9  years,  working  at  the  Atlanta  ARTCC 
area  containing  the  corresponding  actual  sectors.  They  were  current  and  qualified  on  those 
sectors.  Nine  other  controllers  (three  each  week)  from  Gther  areas  of  the  same  ARTCC  were 
paired  with  the  R-controllers  to  serve  as  D-controllers.  In  addition,  six  Atlanta  Center  controllers 
(two  each  week)  worked  at  the  two  non-test  sectors  during  the  simulation.  Former  controllers 
assisted  at  the  non-test  sectors  and  the  aircraft  termination  sector.  The  sectors  and  personnel  are 
shown  in  Figure  6. 

The  test  sector  controllers  were  acquainted  with  features  of  the  simulation  prior  to  their  first 
session  so  that  irrelevant  differences  would  not  interfere  with  their  performance.  In  this  initial 
briefing  they  were  told,  for  example,  about  sector  ‘frequencies,”  rates  of  simulated  climb  and 
descent,  and  the  operation  of  the  aircraft  termination  sector. 

2.9.2  Simulation  Pilots 

Thirty  trained  non-pilot  personnel  participated  in  the  role  of  pilots.  A  former  commercial  airlines 
pilot  o  oplied  additional  instruction  for  this  simulation.  Seven  pilot  positions  were  established  for 
Dublin  High  and  Macon  High,  and  ten  for  Sinca  Low.  Six  pilots  staffed  the  non-test  sectors. 

2.9.3  Operations  Observers 

The  operations  observers  were  three  supervisory  controllers  from  Atlanta  ARTCC  who  were 
currently  specializing  in  areas  other  than  the  one  containing  the  test  sectors.  Each  had  at  least 
five  years  of  experience  in  air  traffic  control  work,  with  a  mean  of  13.3  years.  These  controllers 
were  familiar  with  the  simulation  from  having  assisted  with  scenario  development,  and  par¬ 
ticipated  in  the  initial  briefings  of  the  test  sector  controllers.  They  also  participated  in  the 
development  of  the  questionnaires. 

Prior  to  the  first  simulation  session,  the  operations  observers  were  randomly  assigned  to  a  sector, 
where  they  remained  throughout  all  three  weeks  of  simulation  testing.  They  reviewed  definitions 
of  the  events  they  were  to  record  just  prior  to  the  beginning  of  the  first  simulation  session  and  any 
questions  about  the  event  categories  were  answered  at  that  time.  These  measures  were  taken  to 
assure  consistent  data  recording. 


2-15 


Figure  6.  Simulation  Sectors  and  Personnel 


2.9.4  Communications  Observers 

Three  trained  observers  in  the  NSSF  Pilot  Laboratory  listened  through  earphones  to  all  of  the 
communications  within  the  pilot  laboratory  and  between  the  pilot  and  controller  laboratories. 
Each  was  assigned  at  random  to  a  sector  prior  to  the  tests,  and  remained  there  throughout  all  of 
the  simulation  testing. 

2.10  Procedure 

2.10.1  Test  Sessions 

The  simulated  air  traffic  was  to  be  controlled  as  it  would  have  been  in  an  actual  operational 
environment.  The  R-controllers  handled  all  of  the  communications  with  pilots,  while  all  ground- 
to-ground  communications  were  to  be  handled  by  the  D-controllers. 

The  pilots  in  each  sector  made  scripted  calls  to  the  controller  through  headset  microphones.  The 
message  content  and  time  of  these  scripted  pilot  calls  were  provided  on  the  pilots’  visual  display 
units.  The  pilots  were  instructed  to  respond  rapidly,  but  to  refrain  from  speaking  when  another 
voice  could  be  heard.  They  controlled  the  simulated  aircraft  by  entering  commands  on  a  key¬ 
board. 

It  was  anticipated  that  the  simulated  air  traffic  would  build  gradually  to  its  assigned  level,  with 
higher  levels  requiring  more  time.  Thus,  lengthier  simulation  sessions  were  provided  at  the  higher 
traffic  loads  so  that  sessions  with  all  three  traffic  loads  would  be  at  their  assigned  levels  for 
approximately  the  same  time. 

The  simulation  sessions  ranged  in  duration  from  61  min.  to  82.5  min.,  with  a  scheduled  half  hour 
break  between  sessions.  Sessions  corresponding  to  the  four  delay  combinations  lasted  for  means 
of  69.33  min.  (Today),  70.00  min.  (VSCS),  70.17  min.  (Today  +  Sat),  and  72.2  min.  (VSCS  +  Sat). 
Sessions  corresponding  to  the  three  communication  workload  levels  lasted  for  means  of  66.62 
min.  (medium),  70.75  min.  (high),  and  72.7  min.  (very  high). 

2.10.2  Pilot  Contention 

Realistic  contention  among  pilots  for  a  busy  communication  channel  and  hence  realistic  estimates 
of  pilot-controller  step-ons  requires  a  one-to-one  correspondence  between  pilots  and  aircraft. 
The  assignment  of  several  pilots  to  each  sector  was  intended  to  increase  contention  for  access  to 
the  communications  channels.  Seven  pilots  were  assigned  to  both  Dublin  High  and  Macon  High; 
ten  were  assigned  to  Sinca  Low.  In  addition,  pilots  in  the  same  sector  were  assigned  to  alternate 
seats  to  mask  visual  cues  to  communications  channel  use. 

Aircraft  entering  a  sector  were  assigned  to  the  pilot  with  the  fewest  aircraft.  Up  to  twenty, 
seventeen  and  nineteen  aircraft  per  five  minute  interval  were  handled  in  Dublin  High,  Macon 
High  and  Sinca  Low,  respectively.  Therefore,  pilots  in  Dublin  and  Macon  were  responsible  for  no 
more  than  three  aircraft  simultaneously,  while  those  in  Sinca  were  responsible  for  no  more  than 
two3. 

3  These  maximum  aircraft-to-piiot  ratios  were  obtained  by  dividing  the  highest  number  of  aircraft  in  each  sector 
per  five  minute  interval  by  the  number  of  pilots  in  the  sector.  Average  aircraft-to-pilot  ratios  can  be  obtained 
by  dividing  the  mean  traffic  density  values  given  in  Table  1  by  the  corresponding  number  of  pilots  per  sector. 
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2.10.3  Data  Collection. 

Communications  between  the  pilot  and  controller  positions  were  monitored  from  the  NSSF  Pilot 
Laboratory  by  the  three  communications  observers,  each  of  whom  listened  through  headphones 
to  communications  in  a  different  one  of  the  three  test  sectors  (Figure  7).  The  communications 
observers  recorded  pilot-pilot,  pilot-controller,  and  controller-pilot  step-ons.  The  location  of 
their  headphone  connections  enabled  them  to  hear  all  of  the  communications  in  their  respective 
sectors  regardless  of  whose  microphones  were  keyed.  Taped  voice  recordings  were  later  used  to 
verily  their  data. 

Their  location  in  the  NSSF  Pilot  Laboratory  caused  the  communications  observers  to  hear  (and 
tape)  controller  communications  following  the  relevant  delay  and  pilot  communications  without 
the  delay.  For  this  reason  some  pilot  calls  were  heard  to  arrive  first  when  the  controller  was  in  fact 
the  first  to  transmit.  The  controller-pilot  and  pilot-controller  step-on  categories  were  accordingly 
merged  into  a  single  “pilot-controller/controller-pilot”  step-on  category. 

The  communications  observers  in  the  NSSF  Pilot  Laboratory  could  hear  all  of  the  step-ons,  but 
the  step-ons  recorded  by  the  operations  observers  in  the  Enroute  Laboratory  were  limited  to 
controller-pilot  step-ons  caused  by  the  controller  keying  the  mike  during  the  reception  of  the 
pilot  message,  and  pilot/pilot  step-ons  occurring  when  the  controller’s  mike  was  not  keyed.  The 
operations  observers  could  not  hear  pilot-controller  or  pilot-pilot  step-ons  when  the  controller’s 
mike  was  keyed  or  controller-pilot  step-ons  caused  by  the  controller  keying  the  mike  during  the 
delay  interval  of  an  incoming  pilot  message.  For  this  reason  the  number  of  step-ons  recorded  by 
the  communications  observers  in  the  Pilot  Laboratory  may  differ  from  those  recorded  by  the 
operations  observers  in  the  Enroute  Laboratory.  The  accurate  counts  of  step-ons  made  in  the 
Pilot  Laboratory  are  used  for  the  analysis  of  the  delay  effects. 

In  the  Enroute  Laboratory  an  operations  observer  was  positioned  in  front  of  each  sector, 
observing  the  PVD  and  listening  to  the  communications  channel  used  by  the  radar  controller 
(Figure  8).  The  operations  observers  recorded  the  number  of  callbacks  each  step-on  required 
and  verbal  mistakes  made  by  the  R-controller.  Mistakes  were  classified  according  to  whether  they 
were  corrected  in  the  same  transmission  or  not.  Step-ons  were  classified  by  the  number  of 
additional  calls  needed  to  communicate  the  blocked  message.  A  step-on  could  require  zero,  one 
or  two  callbacks.  Pilot-pilot  step-ons  required  fewer  callbacks  than  would  be  needed  in  actual 
operations  because,  as  noted,  the  simulation  did  not  produce  blocking.  For  this  reason  these  data 
underestimate  the  real  world  number  of  additional  calls  caused  by  step-ons.  They  are  used  as  an 
indication  of  the  additional  communications  workload  caused  by  step-ons  in  the  simulation. 
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Figure  7.  Communications  Observer  and  Data  Recording  Equipment 


Certain  data  were  automatically  recorded  by  the  computer  system.  These  included  the  number 
of  separation  infringements  and  counts  of  aircraft  handled.  Separation  infringements  were 
defined  for  the  study  as  occurring  when  two  aircraft  had  a  horizontal  separation  of  less  than  five 
miles  and  a  vertical  separation  of  less  than  1950  ft.  if  either  aircraft  was  above  Flight  Level  290 
(29,000  ft.  altitude),  or  950  ft.  vertical  separation  if  both  were  at  or  below  Flight  Level  290.  This 
condition  had  to  persist  continuously  for  24  secs,  for  one  infringement  to  be  counted  for  a  pair  of 
aircraft.  Both  of  the  conflicting  aircraft  also  needed  to  be  within  one  or  more  test  sectors  for  the 
infringement  to  be  counted.  Aircraft  were  counted  as  “handled”  following  contact  between  pilot 
and  controller. 

Questionnaires  were  completed  immediately  following  each  session  by  both  controllers  and 
operations  observers.  These  instruments  were  used  to  record  a  variety  of  subjective  impressions 
of  the  preceding  session,  including  perceptions  of  controller  stress,  the  amount  of  attention  or 
concentration  required  during  the  preceding  session,  smoothness  of  traffic  flow,  realism  of  the 
simulation,  and  other  information.  The  questionnaires  are  presented  in  Appendix  D. 
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3.  Results 


3.1  Overview 

The  primary  results  of  interest  were  the  effects  of  delay  on  communications  disruption,  par¬ 
ticularly  step-ons.  Effects  of  delay  on  stress,  verbal  mistakes,  and  system  performance  probably 
result  from  step-ons,  so  the  analysis  of  these  effects  is  likely  to  be  less  sensitive  than  the  analysis 
of  step-ons. 

The  test  of  system  performance  degradation  depends  to  a  large  extent  on  our  ability  to  detect 
changes  in  the  frequency  of  separation  infringements.  These  infringements  occur  very  infrequent¬ 
ly  in  the  real  world.  It  would  take  a  very  major  change  in  the  frequency  of  infringements  due  to 
communication  delay  to  be  statistically  significant. 

Changes  in  stress  are  readily  measurable  but  in  this  simulation  the  level  of  stress  and  workload  is 
due  directly  to  aircraft  handled  and  is  only  an  indirect  side-effect  of  communications  disruptions, 
particularly  step-ons. 

Controller-pilot  and  pilot-controller  step-ons  may  result  from  any  type  of  communication  delays 
if  they  are  of  sufficient  magnitude.  Step-ons  are  likely  to  result  in  losses  of  information  which 
necessitate  repeated  transmissions.  They  may  thereby  subject  the  controller  to  increased 
workload  and  brief  lapses  or  redirection  of  attention.  Also,  critical  communications  may  be  lost 
when  there  is  insufficient  time  for  the  blocked  information  to  be  retransmitted. 

Communications  Disruption 

VSCS  -  No  statistically  significant  increases  in  communications  disruptions  were  found  which 
could  be  attributed  to  simulated  VSCS  delays. 

Satellite  -  Step-ons  -  More  pilot-controller/controller-pilot  step-ons  were  recorded  for  satellite 
conditions  than  non-satellite  conditions.  The  greatest  increases  were  between  the  satellite  and 
non-satellite  conditions  at  the  highest  (very  high)  workload;  these  were  statistically  significant. 
The  smaller  increases  in  step-ons  found  between  satellite  and  non-satellite  conditions,  recorded 
at  the  medium  and  high  communications  workload,  approached  significance. 

In  Macon  High  (a  sector  which  had  much  lower  communications  rates  at  all  workload  levels  than 
Sinca  Low  and  Dublin  High)  there  were  no  significant  differences  in  step  on  rate  between  satellite 
and  non-satellite  conditions. 

Satellite-Verbal  mistakes  -  No  significant  differences  in  verbal  mistakes  were  found  between 
satellite  and  non  satellite  conditions. 

Controller  Stress 

VSCS  and  Satellite  -  No  statistically  significant  differences  were  found  in  measures  of  stress, 
effort,  or  attention  between  any  of  the  delay  conditions,  VSCS  or  Satellite. 

System  Performance 

VSCS  and  Satellite  -  No  statistically  significant  differences  were  found  in  measures  of  separation 
infringements  or  numbers  of  aircraft  handled. 


The  following  sections  describe  the  results  of  the  statistical  analyses  and  the  constraints  and 
limitations  on  their  interpretation.  The  results  of  the  measures  of  communications  disruptions 
and  system  performance  are  described  first,  followed  by  controller  and  operations  observer 
ratings  of  controller  stress.  Table  6  presents  the  abbreviations  used  in  the  following  tables  and 
appendices.  Tables  7,  8,  and  9  show  the  results  for  the  main  dependent  variables.  Table  8  shows 
effects  of  delay.  Table  9  shows  the  effect  of  communications  workload.  Appendix  C  contains  a 
discussion  of  the  statistical  checks  to  ensure  the  validity  of  the  simulation.  A  discussion  of  the 
analysis  procedures  can  be  found  in  Appendix  E.  Appendix  F  contains  the  output  of  the 
statistical  program  upon  which  the  description  of  the  results  is  based. 

Table  7  shows  the  statistical  significance  of  the  regression  analyses.  The  independent  variables  and 
interactions  of  interest  are  in  the  column  headings  of  the  table;  the  dependent  variables  are  in  the 
row  headings.  The  cells  contain  values  each  representing  the  probability  that  a  difference  in  a 
particular  dependent  variable  is  due  to  chance  rather  than  a  systematic  effect.  For  example,  the 
.0001  at  the  intersection  of  Load  and  CM  indicates  the  probability  that  differences  in  corrected 
verbal  mistakes  among  the  three  communications  load  conditions  are  due  to  chance.  The  two 
columns  under  Model  indicate  the  probability  that  all  differences  in  the  analysis  of  a  particular 
dependent  variable  are  due  to  chance  (Prob)  and  the  proportion  of  the  variation  in  the  data  for 
a  particular  dependent  variable  that  is  captured  by  all  of  the  independent  variables  in  the  analysis 
(R-Sqr).Tinted  cells  contain  probability  values  that  indicate  statistically  significant  differences. 
The  choice  of  p  <  .01  for  significance  reflects  the  large  number  of  comparisons  that  are  made  in 
each  regression  analysis. 
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Table  7  Regression  Analyses  of  Objective  Measures 
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Table  8.  LS  Means  of  Objective  Measures  by  Delay 


Table  9.  Means  of  Objective  Measures  by  Communications  Load 


3.2  Limitations  on  the  Results 

A  number  of  factors  inherent  in  the  simulation  limited  the  sensitivity  of  the  study.  Thus  small 
differences  which  approached  significance  in  the  study  might  have  been  reached  the  threshold  of 
statistical  significance  in  a  longer  study. 

As  noted  in  section  2.8.1  pilot-pilot  step-ons  in  the  simulation  did  not  produce  blocking.  For  this 
reason  the  data  underestimate  the  real  world  number  of  additional  call-backs  caused  by  step-ons. 

When  a  pilot-pilot  step-on  occurs  in  the  real  world  both  transmissions  are  blocked  and  the 
controller  hears  a  distracting  squeal.  Lack  of  a  squeal  may  have  reduced  the  stress  experienced 
by  the  controllers. 

The  sensitivity  of  the  analysis  of  separation  infringements  is  restricted  by  the  number  of  sessions 
possible  in  the  study.  In  the  real-world  so  few  infringements  occur  that  meaningful  statistical 
analysis  is  very  difficult. 

3.3  Communications  Disruptions 

3.3.1  Step-ons 

Data  for  the  analyses  of  step-ons  were  recorded  by  the  communications  observers.  Retransmis¬ 
sion  (callback)  data  was  recorded  by  the  operations  observers. 

Pilot-pilot  step-ons  -  Multiple  regression  analyses  found  no  statistically  significant  effects  of  delay 
on  pilot-pilot  step-ons.  Examination  of  the  LS  (least  square)  means  did  not  indicate  increased 
pilot-pilot  step-ons  in  any  of  delay  conditions  (see  Table  10).  No  delay  effects  were  expected 
because  calls  made  by  one  pilot  were  heard  by  another  with  no  delay. 


Table  10.  Pilot-Pilot  Step-Ons  (LS  Means) 


|  Delay 

Load 

Today 

vscs 

Today + Sat 

VSCS  +  Sat 

Medium 

0.89 

0.52 

1.37 

0.11 

High 

1.33 

1.61 

2.02 

1.21 

Very  High 

2.02 

2.28 

1.61 

2.18 

Controller-pilot/pilot  controller  step-ons  -  The  regression  analysis  indicated  that  there  was  a 
statistically  significant  effect  of  delay  and  a  statistically  significant  delay  by  communications 
workload  interaction.  The  effect  of  subject  was  statistically  non-significant.  A  Tukey  HSD  analysis 
was  conducted  to  compare  the  individual  cells.  It  revealed  that  there  were  significantly  more 
controller-pilot/pilot  controller  step-ons  in  either  of  the  very  high  communication  workload 
satellite  delay  conditions  than  in  either  of  the  very  high  communication  workload  non-satellite 
delay  conditions  (see  Figure  9). 

An  initial  regression  analysis  found  no  statistically  significant  differences  (p>.25)  for  com¬ 
parisons  at  medium  and  high  communication  workload  of  all  satellite  vs  non-satellite  conditions. 
A  second  analysis  suggested  by  the  data  compared  both  of  the  satellite  conditions  with  both  of 
the  non-satellite  conditions,  and  dropped  comparisons  at  very  high  workload  .  This  procedure, 
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which  increased  statistical  power,  led  to  differences  that  approached  statistical  significance  at 
p  =  .04.  Taken  as  a  whole  the  analyses  confirm  neither  the  presence  nor  the  absence  of  a 
statistically  significant  effect  of  satellite  delay  at  the  medium  and  high  levels  of  communications 
workload. 

VSCS  -  The  differences  between  Today  and  VSCS,  and  between  Today  +  Satellite  and 
VSCS  +  Satellite,  were  attributable  only  to  chance  variation  in  the  subjects’  performances,  rather 
than  any  delay  effects .  P-diffs,  which  show  the  probability  that  the  difference  between  a  pair  of 
conditions  is  due  to  chance,  are  presented  in  Table  11  for  all  of  the  comparisons  between 
conditions  with  VSCS  and  conditions  without  VSCS.  The  P-diffs  show  that  there  is  no  suggestion 
of  a  significant  difference  between  any  of  the  conditions  with  VSCS  and  the  conditions  without 
VSCS. 

At  the  very  high  level  of  communications  workload: 

•  The  Today  delay  resulted  in  an  LS  mean  of  3.90  step-ons,  the  VSCS  delay 
resulted  in  an  LS  mean  of  3.29  step-ons. 

•  The  Today  +  Sat  delay  resulted  in  an  LS  mean  of  9.75  step-ons. 

•  The  VSCS  +  Satellite  delay  resulted  in  an  LS  mean  of  9.07  step-ons. 

At  the  very  high  level  of  communications  workload: 

•  The  addition  of  satellite  delays  to  the  Today  delay  condition  resulted  in  a  150% 
increase  in  controller-pilot/pilot-controller  step-ons. 

•  When  added  to  the  VSCS  delay  condition,  satellite  delays  resulted  in  a  176%  in¬ 
crease  in  this  type  of  step-on. 

At  the  medium  and  high  levels  of  communications  workload: 

•  The  increase  in  step-ons  from  non-satellite  to  satellite  delays  evident  in  Figure  9 
approached  significance,  but  cannot  be  confirmed  as  statistically  significant 
given  the  limited  sample  size  possible  in  this  study. 


0  Today  B  VSCS  Q  Today+Sat  jg  VSCS+Sat 


Table  11.  Chance  Probability  Values 
for  Steps  with  and  without  VSCS. 


Today  vs. 

VSCS 

Today -i- Sat  vs. 

VSCS + Sat 

Dublin 

.98 

.81 

Sector 

Macon 

.76 

.18 

Sinca 

.95 

.37 

Communication 

Medium 

.91 

.65 

Workload 

High 

.37 

.81 

Very  High 

.68 

.67 

|  Overall 

.83 

.90 

Step-ons  as  a  function  of  communications  activity  -  The  design  used  in  this  study  manipulated 
communications  work  load  by  changing  air  traffic  levels.  The  assumption  used  was  that  high  traffic 
loads  would  result  in  high  communications  workloads.  This  was  true;  however,  the  levels  of  traffic 
(medium,  high,  very  high)  generated  different  levels  of  communications  activity,  i.e.,  microphone 
push-to-talk  keying  for  the  different  sectors  for  the  same  levels  of  traffic.  In  particular  much  lower 
levels  of  communications  activity  or  workload  were  found  for  the  Macon  High  simulation  than 
for  Sinca  Low  and  Dublin  High  (see  Table  2). 

The  number  of  push-to-talks  per  session  was  used  as  an  analog  of  communications  workload  in 
sector  analyses  of  controller-pilot/pilot-controller  step-ons.  Before  discussing  the  sector  data  it  is 
important  to  examine  the  relationship  between  push-to-talks  and  step  ons  for  all  sectors  com¬ 
bined.  Figure  10  is  a  scatter  plot  where  the  number  of  step-ons  is  plotted  against  the  push-to-taiks 
per  session.  The  triangles  represent  the  non-satellite  delay  conditions,  the  squares  the  satellite 
delay  conditions. 
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Non-Satellite  °  Satellite 


At  almost  all  levels  of  push-to-talk  activity  the  step-on  rates  appear  to  be  higher  for  the  satellite 
conditions  than  for  the  non-satellite  conditions.  Figure  1 1  is  the  same  data  aggregated  into  three 
levels  of  push  to  talk;  in  each  sub-group  the  satellite  delay  step-ons  are  approximately  two  to  three 
times  as  high  as  the  non-satellite  step-ons.  Examination  of  Figures  12 , 13  and  14  indicate  that  this 
relationship  is  found  both  for  Sinca  Low  and  Dublin  High,  but  not  for  Macon  High.  This  may  be 
because  the  simulated  traffic  loads  for  Macon  High  produce  much  less  communications  activity 
than  those  for  the  other  two  sectors. 

Re-transmissions  (Callbacks)  -  The  operations  observers’  counts  of  step-ons  were  also  subjected 
to  a  multiple  regression  analysis.  The  effect  of  delay  on  the  number  of  step-ons  approached 
statistical  significance  (p  =  .026)  as  did  the  interaction  of  delay  with  communications  workload 
(p  =  .019).  The  effect  of  communications  workload  and  variation  due  to  sector  were  statistically 
significant.  Table  8  shows  more  step-ons  due  to  satellite  than  non-satellite  delay  conditions. 
While  these  data  could  differ  from  those  collected  in  the  Pilot  Laboratory,  as  explained  in  section 
2.10.3,  the  results  are  generally  consistent  (in  any  event  the  Pilot  Laboratory  data  are  to  be 
considered  the  more  accurate). 

These  data  are  used  to  assess  the  additional  communications  workload  caused  by  step-ons  in  the 
simulation.  One  of  the  likely  effects  of  step-ons  is  the  need  for  the  sender  to  retransmit 
information  which  might  have  been  lost  due  to  the  step-on.  At  worst,  step-ons  can  generate 
callbacks  that  would  themselves  be  blocked  by  further  step-ons.  Three  categories  of  step-on  were 
recorded  by  the  operations  observers  depending  on  the  number  of  additional  calls  made. 

The  mean  numbers  of  step-ons  requiring  zero,  one,  and  two  additional  calls  were  1.67,  1.70,  and 
0.06,  respectively,  per  session  and  per  sector.  These  results  mean  that  48.7%  required  no 
additional  calls,  and  49.6%  of  the  step-ons  required  one  additional  call.  Only  1.7%  of  the  step-ons 
required  two  additional  calls.  Thus,  on  the  average,  a  stepped-on  message  required  0.53  addition¬ 
al  calls.  As  noted  in  section  2.10.3,  the  number  of  call  backs  required  due  to  blocking  during  the 
simulation  were  probably  less  than  those  which  would  occur  in  the  real-world  because  the 
simulation  did  not  include  blocking  during  pilot-pilot  step-ons. 

3.3.2  Corrected  and  Uncorrected  Verbal  Mistakes. 

Operations  observers  recorded  the  R-controllers’  corrected  and  uncorrected  verbal  mistakes. 
Neither  showed  a  statistically  significant  effect  of  delay,  regardless  of  whether  the  mistakes  were 
analyzed  separately  or  together.  Figure  15  depicts  the  results  for  corrected  mistakes.  The  values 
corresponding  to  Figure  15  are  shown  in  Table  12.  The  regression  analysis  of  the  corrected 
mistake  data  shows  that  communications  workload,  sector,  and  subject  all  affected  this  dependent 
variable. 

Whereas  corrected  verbal  mistakes  were  those  corrected  in  the  same  transmission  as  the  mistake, 
uncorrected  mistakes  included  those  corrected  in  a  later  transmission  as  well  as  those  that  were 
not  corrected.  The  effect  of  communications  workload  on  uncorrected  mistakes  approached 
significance  (p  =  .025).  However,  there  were  too  few  uncorrected  mistakes  (only  36  in  the  entire 
study)  to  anticipate  statistically  significant  results. 
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Figure  13.  Steps  by  Communications  Activity  Grouping  tor  Macon  High 
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Figure  15.  Mean  Corrected  Mistakes  per  Session 


Table  12.  Mean  Corrected  Mistakes  per  Session 


Delay 

Load 

Today 

vscs 

Today + Sat 

VSCS  +  Sat 

Medium 

0.78 

1.67 

1.50 

1.37 

High 

3.00 

2.66 

3.17 

4.00 

Very  High 

4.83 

5.44 

4.33 

4.00 

3.4  Estimates  of  Controller  Stress 

Two  items  measuring  controller  stress  were  included  in  the  questionnaires  filled  out  by  the 
R-controllers  and  operations  observers.  One  item  asked  the  controllers  to  rate  the  amount  of 
stress  that  they  experienced  in  the  preceding  simulation  session  and  the  observers  to  rate  the 
amount  of  stress  the  controllers  appeared  to  experience;  the  other  item  asked  both  the  R-control¬ 
lers  and  the  operations  observers  to  rate  the  amount  of  attention  and  concentration  required  to 
control  the  air  traffic  during  the  preceding  session.  Delay  showed  no  statistically  significant 
effects  on  the  responses  to  these  questions  (p  >  .14),  except  for  a  marginal  delay  by  load  interac¬ 
tion  (p  =  .022),  apparently  due  to  less  stress  reported  in  the  Today  +  Sat.  condition  at  very  high 
load.  Stress  and  attention  were  found  to  increase  significantly  with  communications  workload 
(pc.OOOl).  There  were  also  significant  differences  in  stress  ratings  associated  with  differences 
between  sectors  (p  <  .0001)  and  subjects  (p  =  .003)  in  the  coi.  roller  self  report  ratings  but  not  in 
the  observer’s  reports  (p  =  .095  for  sectors  and  p  =  .040  for  subjects).  Attention  was  found  to  differ 
between  sectors  (p  =  .0005)  and  subjects  (pc.OOOl),  but  only  as  rated  by  the  controllers  ;  the 
observer  ratings  differed  marginally  between  sectors  (p  =  .010).  Figures  16-19  and  Tables  13-16 
present  the  stress  and  attention  results. 
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Figure  18.  Mean  Attention  Required  as  Reported  by  Controllers 
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Table  13.  Stress  as  Reported  by  Controllers 


jj  Delay 

Load 

Today 

VSCS 

Today + Sat 

VSCS  +  Sat 

Medium 

1.56 

1.17 

1.50 

1.17 

High 

2.25 

2.33 

2.17 

2.67 

Very  High 

2.71 

2.89 

3.33 

3.00 

Mean  ratings  on  a  5-pt.  scale:  1  =  no  stress  and  5  =  excessive  stress. 


Table  14.  Stress  as  Reported  by  the  Operations  Observers 


Delay 

Load 

Today 

VSCS 

Today + Sat 

VSCS + Sat 

Medium 

1.00 

1.00 

1.00 

1.00 

High 

1.22 

1.57 

1.67 

1.67 

Very  High 

2.67 

2.89 

2.17 

2.67 

Mean  ratings  on  a  5-pt.  scale:  1  =  no  stress  and  5  =  excessive  stress. 


Table  15.  Attention  Required  (Reported  by  Controllers) 


Delay 

Load 

Today 

VSCS 

Today+Sat 

VSCS  +  Sat 

Medium 

1.56 

1.17 

1.50 

1.50 

High 

2.25 

2.33 

2.00 

2.50 

Very  High 

2.86 

2.89 

3.00 

3.17 

Mean  ratings  on  a  4-pt.  scale:  1  =  Little;  2  =  Moderate;  3  =  High;  4=Too  High. 


Table  16.  Attention  Required  (Reported  by  Observers) 


Delay 

Load 

Today 

VSCS 

Today+Sat 

VSCS + Sat 

Medium 

1.00 

1.00 

1.00 

1.00 

High 

2.00 

2.00 

1.88 

2.17 

Very  High 

3.00 

2.67 

2.83 

3.00 

Mean  ratings  on  a  4-pt.  scale:  1  =  Little;  2  =  Moderate;  3  =  High;  4  =  Too  High. 


Two  items  related  to  stress  asked  the  R-controllers  to  indicate  the  importance  of  various  sources 
of  difficulty  in  the  preceding  session  and  the  operations  observers  to  rate  any  tendency  toward 
overcontrol  in  the  performances  of  the  R-controllers. 

Sources  and  Levels  of  Difficulty  -  Figures  20-22  illustrate  the  rated  importance  of  various 
sources  of  difficulty  for  each  type  of  delay  and  communications  workload  level. 
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Figure  20.  Mean  Response  to  "Sources  of  Difficulty"  Question:  Medium  Load 
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Figure  21. Mean  Response  to  "Sources  of  Difficulty"  Question:  High  Load 
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No  systematic  effects  of  delay  are  evident  in  these  figures.  The  corresponding  values  are  provided 
in  Table  17. 

Overcontrol  -  The  operations  observers  reported  little  tendency  to  overcontrol  (act  too  far  ahead 
of  the  aircraft)  among  the  R-controllers.  No  cases  of  extensive  overcontrol  were  reported.  Three 
observations  of  “some  overcontrol”  were  recorded  following  sessions  with  Today,  VSCS,  and 
VSCS  +  Sat  delays.  The  remaining  78  observations  indicated  no  tendency  to  overcontrol. 

3.5.  System  Performance 

Indications  of  system  performance  included  separation  infringements,  number  of  aircraft  handled 
and  rated  smoothness  of  the  flow  of  air  traffic.  Separation  infringements  were  said  to  occur  when 
two  aircraft  violated  the  separation  requirements  used  for  this  study.  (A  precise  definition  can  be 
found  in  Section  2.10.3).  Figure  23  depicts  the  number  of  separation  infringements  per  session 
for  each  delay  and  communication  workload  condition.  The  values  for  this  figure  are  provided  in 
Table  18.  No  trend  due  to  delay  is  evident  in  this  data.  The  number  of  infringements  during  each 
session  and  in  each  sector  is  found  in  Appendix  G. 

The  infringements  at  the  two  very  high  communication  workloads  were  higher  than  under  any 
other  conditions.  Because  there  were  only  38  infringements  recorded  during  the  entire  study  the 
data  were  aggregated  into  satellite  and  non-satellite  delay  conditions  (see  Figure  24).  This  more 
clearly  reveals  the  lack  of  effect  of  delay  on  infringements  found  in  this  study.  A  regression 
analysis  confirms  the  lack  of  statistically  significant  effects  due  to  delay.  Communications 
workload  produced  a  marginally  significant  effect  on  separation  infringements  (p  =  .013). 

Statistically  significant  effects  would  have  required  a  striking  increase  in  the  number  of  infringe¬ 
ments  recorded  because  the  small  number  of  infringements  observed  reduced  the  power  of  the 
statistical  test.  Even  with  the  most  aggregate  analysis,  detection  of  a  significant  difference 
between  satellite  and  non-satellite  conditions  would  require  twice  as  many  in  one  condition  as  in 
the  other.  For  example,  if  26  separation  infringements  occurred  in  the  satellite  conditions  and 
twelve  in  the  non-satellite  conditions,  the  difference  would  be  just  significant  at  the  .05  level. 

The  38  separation  infringements  was  much  higher  than  expected  on  the  basis  of  Atlanta  Center 
records  (zero  or  one  in  the  entire  simulation,  as  noted  in  Section  1.0).  Irregularities  in  the 
simulation,  including  pilot  "miskeying,"  caused  most  or  all  of  these  infringements.  For  example, 
if  a  pilot  transferred  an  aircraft  to  an  incorrect  sector  “frequency”  the  aircraft  would  move 
erratically  across  a  test  sector  and  could  not  be  removed  from  the  PVT)  until  it  entered  a  different 
test  sector.  Problems  with  pilot  scripting  were  responsible  for  other  irregularities. 
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an  Response  to  “Sources  ol  Difficulty"  Question 
Communications  Workload  and  Delay 
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Rating  of  impoi  tancc  on  0-9  scale:  0  =  Not  Important;  9  =  Very  Important. 
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Figure  23.  Mean  Separation  Infringements  vs.  Delay  and  Communications  Workload 
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Table  18.  Mean  Separation  Infringements  (per  Session) 


I  DELAY  1 

LOAD 

Today 

VSCS 

Today  +  Sat 

VSCS  +  Sat 

Medium 

1.00 

1.00 

0.50 

2.00 

High 

0.67 

2.00 

1.00 

0.00 

Very  High 

2.50 

3.33 

3.50 

2.50 

Aircraft  Handled  -  Figure  25  depicts  the  mean  number  of  aircraft  handled  for  each  delay  and 
workload  condition.  Table  19  provides  the  corresponding  values.  A  regression  was  conducted  on 
the  number  of  aircraft  handled  each  session  in  each  test  sector.  Variation  due  to  communications 
workload  and  sector  were  statistically  significant.  No  significant  differences  due  to  delay  nor 
interactions  with  delay  were  found,  although  the  interaction  of  delay  and  communication 
workload  was  nearly  significant  (p  =  .026). 


Table  19.  Aircraft  Handled  (Mean  Aircraft  per  Session) 


!  DELAY 

LOAD 

Today 

VSCS 

VSCS  +  Sat 

Medium 

34.67 

35.17 

33.67 

32.33 

High 

45.56 

45.67 

47.33 

45.00 

48.17 

55.00 

53.50 

56.50 

Smoothness  of  TVaffic  Flow  -  The  operations  observers  were  asked  to  rate  the  extent  of  any 
disruption  in  air  traffic  flow  in  the  preceding  session.  None  indicated  severe  disruption  in  any 
session.  Four  cases  of  “moderate  disruption”  were  recorded,  one  in  each  delay  condition.  The 
remaining  77  observations  indicated  no  disruption. 
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4.  Discussion 


The  study  was  designed  to  test  three  hypotheses: 

•  System  performance  is  degraded  with  increased  delay. 

•  Controller  stress  increases  with  delay. 

•  Step-ons  increase  with  delay. 

Delay  was  not  found  to  affect  measures  of  stress  or  of  system  performance.  The  measures  of  stress 
and  performance  were  found  to  vary  with  communications  workload  (except  in  cases  where  there 
were  too  few  events  recorded  to  permit  a  satisfactory  statistical  analysis).  Only  step-ons  were 
found  to  increase  with  delay  and  then  only  for  the  long  delays  characteristic  of  satellites  at  the 
highest  communications  workload  level. 

In  the  real  world  step-ons  may  be  distractions  and  are  arguably  a  cause  of  controller  stress,  which 
can  in  turn  affect  system  performance,  i.e.,  step-ons  potentially  increase  workload  and  could  be 
detrimental  to  system  performance.  Loss  of  information  due  to  step-ons  can  also  degrade  system 
performance.  In  the  simulation  the  absolute  number  of  step-ons  was  limited  by  the  exclusion  of 
emergencies  and  weather,  which  would  have  increased  communications  workload  and  thus 
increased  step-ons. 

Therefore  under  all  delay  and  communications  workload  conditions  fewer  step-ons  would  be 
expected  than  in  real  world  operations.  Because  of  this  limitation  in  the  number  of  step-ons  (no 
sector  ever  had  more  than  0.42  step-ons  per  minute)  the  impact  of  step-ons  on  controller  stress 
and  system  performance  may  have  been  limited. 

Limitations  in  the  electronic  aspects  of  the  simulation  (lack  of  a  squeal)  reduced  loss  of  informa¬ 
tion  due  to  pilot-pilot  step-ons,  and  thus  reduced  the  number  of  call-backs  and  irritation  to  the 
controller.  This  also  may  have  the  obscured  the  relationship  between  step-ons  and  the  stress  and 
system  performance  measures. 
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5.  Conclusions 


System  Performance 

•  No  statistically  significant  differences  were  found  in  measures  of  separation  in¬ 
fringements  or  numbers  of  aircraft  handled  between  any  of  the  delay  conditions. 

Communications  Disruptions 

•  No  statistically  significant  increases  in  communications  disruptions  were  found 
which  could  be  attributed  to  simulated  VSCS  delays. 

•  Statistically  significantly  more  pilot-controller/controller-pilot  step-ons  were 
recorded  for  satellite  conditions  than  non-satellite  conditions  at  very  high 
workload. 

•  In  Macon  High  (a  sector  which  had  much  lower  communications  rates  at  all 
workload  levels  than  Sinca  Low  and  Dublin  High)  there  were  no  apparent  dif¬ 
ferences  in  step-on  rate  between  satellite  and  non-satellite  conditions. 

•  No  statistically  significant  differences  in  verbal  mistakes  were  found  between 
satellite  and  non-satellite  conditions. 

Controller  Stress 

•  No  statistically  significant  differences  were  found  in  measures  of  stress,  effort, 
or  attention  between  any  of  the  delay  conditions. 

Even  though  increased  step-ons  were  found  to  be  associated  with  satellite  delay  conditions,  this 
study  did  not  demonstrate  that,  in  and  of  itself,  delay  will  cause  significant  negative  impacts  on 
system  performance.  Evidence  obtained  from  analysis  of  ATC  operations  suggests  that  voice 
communication  difficulties  and  distractions  can  lead  to  operational  errors,  particularly  when  the 
controller  is  handling  large  numbers  of  aircraft. 1  However  the  fact  that  satellite  systems  appear 
to  be  operating  successfully  in  a  limited  number  of  areas  suggests  that  strategies  have  evolved 
which  ameliorate  the  effects  of  increased  step-ons. 

The  question  remains:  Will  the  observed  increase  in  step-ons,  under  satellite  delay  conditions, 
particularly  at  very  high  communication  workload  conditions  translate  into  decreased  real  world 
system  performance?  This  matter  can  best  be  resolved  through  a  separate  field  study  of  systems 
which  include  sectors  with  high  communications  activity  and  use  satellite  communications. 


1  Golaszewski,  R.  An  Analysis  of  Pilot-Controller  Read-Back  Errors.  Journal  of  ATC,  December,  1989 
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Appendix  A 

ATC  Voice  Communications  Delay  Test  Plan 

Draft  Coordination/Approval  Version  10-17-89 
Gerard  Spanier  ACN-140 


1 .  Object 

The  object  of  these  tests  is  to  determine  if  communication  delays  in  Air  Traffic  voice  transmissions 
adversely  impact  voice  communications  performance  beyond  baseline  (today’s)  performance 
under  specific  conditions.  The  specific  conditions  include 


•  Type:  enroute  sectors 

•  Air-to-round  voice  communications 

•  Compliance  to  worst  case  VSCS  specification  delay  values 

•  Inclusion  of  end-to-end  set-up  delays  that  exist  today  or  that  will  exist  in  the 
VSCS  time  frame 

•  Delays  due  to  the  use  of  satellite  links 

•  Communications  under  realistically  high  load  conditions.  Significant 
differences  between  the  controller  performances  under  the  tests  will  be 
analyzed  and  reported. 


2.  Background 

As  part  of  the  modernization  of  the  National  Airspace  System  Air  Traffic  Control  (sub)system, 
the  NAS  plan  requires  a  new  Air  Traffic  Control  voice  communications  system,  the  Voice 
Switching  and  Control  System  (VSCS),  specifically  for  use  in  the  enroute  Area  Control  Facilities 
(ACF).  This  system  is  under  development,  and  may  result  in  different  delay  characteristics  due 
to  design.  Other  improvements  to  the  NAS  may  depend  upon  the  use  of  satellite  communications, 
to  improve  accessibility,  reliability,  cost-benefit,  and  expandability.  Communication  delays  could 
possibly  result  in  reduced  numbers  of  voice  communications,  loss  of  communications  due  to 
multiple  transmitters  in  use  at  the  same  time  on  the  same  frequency  (step-ons),  lost  transmissions 
or  excessive  repetitions  due  to  uncertainties  or  hesitations  from  repeated  transmissions.  Willful 
transmission  by  a  pilot  while  that  same  pilot  knows  someone  else  is  transmitting  is  not  part  of 
these  tests.  The  pertinent  performance  of  current  equipment  which  will  be  utilized  by  the  NAS 
with  the  VSCS  in  the  VSCS  time  frame  (RCE,  etc.)  is  simulated  in  the  tests  to  assure  accuracy  and 
realism. 
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By  agreement  between  all  involved  organizations,  interphone  and  intercom  communications  will 
not  be  analyzed  in  this  set  of  tests.  As  per  the  NAS  Plan,  ATCT  and  FSS  environments  are  not 
utilizing  the  VSCS  as  a  communication  system,  therefore  those  environments  are  not  part  of  this 
set  of  tests.  The  terminal  environment  (TRACONS)  will  not  be  utilizing  the  VSCS,  primarily  due 
to  size  incomparability.  These  three  environments  are  not  expected  to  require  the  use  of  satellite 
communications,  so  they  are  not  included  in  the  tests.  The  VSCS  will  be  used  for  voice  com¬ 
munications  in  terminal  sectors  within  an  ACF,  in  accordance  with  the  current  NAS  System 
Design  calling  for  TRACON  consolidation.  This  environment  will  not  be  a  part  of  this  set  of  tests, 
due  to  scenario  and  simulation  limitations.  The  use  of  alternate  communications  paths,  including 
satellite  paths,  as  part  of  the  back-up  and  contingency  requirements  of  the  NAS,  is  not  a  question 
to  be  answered  by  the  tests,  since  the  acceptable  performance  level  for  communications  under 
known  reduced  ATC  performance  during  contingencies  is  not  established  nor  within  the  scope  of 
the  tests. 

3.  Test  Categories 

This  plan  covers  the  following  categories: 

•  Air/ground  and  ground/air  voice  communications 

•  Three  pertinent  and  representative  enroute  sector  types 

-  Low  altitude  high  density  feeder  sector  for  a  major  terminal  facility 
(approach/departure  sector) 

-  High  altitude  sequencing  sector  for  a  low  altitude  sector  and  adjacent  to 
other  high  altitude  sectors 

-  High  altitude  sector  in  the  same  ARTCC,  and  adjacent  to  high  altitude 
sequencing  sector 

•  Non-satellite  transmission  link  in  the  communication  path 

•  One  satellite  transmission  link  (rounu  trip)  in  the  path 

•  Three  levels  of  communications  load 

-  Communication  load  based  on  a  70%-of-peak  sector  traffic  load 

-  Communication  load  based  on  a  90%-of-peak  sector  traffic  load 

-  Communication  load  based  on  a  1 10%-of-peak  sector  traffic  load 

•  Transmitter  set-up,  voice  propagation,  and  other  maximum  delays  as  found  in 

today’s  system 

•  Transmitter  set-up,  voice  propagation,  and  other  maximum  delays  as  specified 

for  the  system  to  be  in  use  during  the  VSCS  time  frame  (99.9  percentile) 


4.  Test  Parameters 

The  test  parameters  represent  those  items  within  the  scope  of  testing  that  will  be  varied  for  each 
of  the  categories,  for  tests  to  be  run.  The  term  ’dead’  time  is  used  below,  and  is  defined  as  the 
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time,  for  someone  listening  on  a  channel,  between  the  end  of  a  pilot’s  transmission  and  the 
reception  of  a  carrier  from  an  enabled  FAA  VHF/UHF  transmitter.  This  is  the  time  a  listener  on 
channel  should  and  would  believe  that  a  channel  is  not  in  use  after  the  end  ef-a  pilot’s  transmis¬ 
sion.  Contention  between  a  controller  and  pilots  for  the  channel  during  this  time  is  a  primary 
cause  of  ’step-ons.’  The  shorter  the  time,  the  less  opportunity  for  ’step-ons.’  Step-ons  due  to 
willful  transmission  while  the  pilot  knows  someone  else  is  speaking  are  not  a  part  of  these  tests. 

4.1  Delays 

4.1.1  Baseline  Delay 

The  baseline  delay  includes  the  set-up  time  delay  that  exists  in  the  present  system  and  is 
represented  by  the  delay  that  occurs  between  the  time  that  a  controller  presses  the  push-to-talk 
(PTT)  switch  and  the  time  when  pilots  are  aware  that  a  PTT  switch  has  been  depressed.  This  time 
is  a  ’dead’  time  as  defined  above,  and,  for  the  Baseline  delay,  consists  of  190  milliseconds  (ms) 
from  PTT  to  transmitter  and  35  ms  for  transmitter  enable.  The  baseline  delay  also  includes  any 
propagation  or  other  delay  through  the  audio  channel  from  the  microphone  of  the  person 
speaking  to  the  earpiece  of  the  person  listening,  with  the  appropriate  timing  relationship. 
’Today’s’  system,  which  is  used  as  the  basis  for  the  delay  values,  contains  essentially  no  propagation 
delay.  Thus,  the  total  delay  to  be  used  as  the  baseline  delay  is  225  ms,  all  due  to  the  set-up  delay. 

4.1.2  VSCS-Time-Frame  Delay 

The  VSCS  delay  has  been  identified  as  follows: 

•  PTT  to  VSCS  output  is  30  ms 

•  VSCS  output  through  the  local  Radio  Control  Equipment  (RCE)  is  50  ms 

•  Local  RCE  through  the  transmission  equipment  (TE)  is  4  ms 

•  TE  output  through  the  remote  RCE  to  transmitter  is  50  ms 

•  Transmitter  enable  is  35  ms 

for  a  total  of  169  ms.  The  VSCS  voice  circuit  has  a  70  ms.  delay  in  voice  transmission,  which  means 
that  the  controller  can  begin  talking  as  early  as  99  ms  after  PTT  without  losing  or  clipping  any  part 
of  the  beginning  of  the  message.  As  a  result,  the  VSCS  delay  without  a  satellite  link  could  be  as 
little  as  169  ms,  including  both  the  169  ms  set-up  delay  and  the  70  ms  voice  delay,  since  these  two 
delays  are  not  serial  (not  additive)  in  the  situation  described  above.  However,  this  value  does  not 
represent  the  (worst  case)  minimum  VSCS-time-frame  delay.  The  same  70  ms.  voice  delay  exists 
in  the  reception  of  pilot  communications  by  the  controller,  but  not  in  the  reception  by  other  pilots. 
This  utterance  delay  at  the  end  of  the  pilot’s  communication  (called  pilot  termination  delay  in  this 
document),  which  chronologically  occurs  before  the  PTT  event,  must  be  added  to  the  VSCS 
set-up  delay,  giving  a  total  delay  of  239  ms.  This  represents  the  total  ’dead’  time  as  defined  above, 
which  makes  it  the  effective  VSCS-time-frame  delay.  For  the  tests,  voice  delay  between  pilots  and 
controllers  is  70  ms,  and  the  simulated  set-up  delay  is  99  ms. 
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4.1 .3  TODAY’S  System  With  Satellite  Delay 

The  satellite  delay  is  nominally  260  ms,  which  is  the  effective  round  trip  delay  encountered  by 
voice  signals  and  control  signals  travelling  from  a  ground  station  to  a  stationary  orbiting  satellite 
approximately  22,000  above  the  earth.  This  delay  exists  in  voice  transmissions  in  both  directions 
between  pilots  and  controllers,  and  in  set-up  control  signals  initiated  by  the  controller’s  PTT 
action.  Adding  260  ms  to  the  225  ms  set-up  delay  of  TODAY  results  in  485  ms.  However,  the 
controller  who  wants  to  PTT  at  the  end  of  a  pilot’s  transmission  must  wait  260  ms  after  the  pilot 
is  finished  before  the  controller  is  aware  that  the  pilot  is  finished.  This  time  must  be  added  to  the 
485  ms,  to  result  in  745  ms  effective  ’dead’  time  for  a  listening  pilot,  during  which  time  the  pilot 
hears  no  indication  that  someone  is  transmitting. 

4.1.4  VSCS  Time-Frame  with  Satellite  Delay 

The  satellite  delay  is  260  ms,  which,  as  above,  is  the  effective  round  trip  delay  encountered  from 
a  ground  station  to  a  stationary  orbiting  satellite  approximately  22,000  miles  above  the  earth.  This 
delay  exists  in  both  transmissions  from  the  controller  to  the  pilots  and  from  the  pilots  to  the 
controller.  Adding  the  satellite  delay  to  the  70  ms  pilot  termination  delay  time  identified  above 
results  in  a  330  ms  delay  in  satellite  pilot  termination,  which  is  not  perceived  by  other  pilots,  but 
affects  the  start  of  the  controller  communications  sequence.  The  controller  can  initiate  the  PTT 
sequence  only  after  the  satellite/pilot  termination  delay.  Following  this  PTT  is  the  delay  of  the 
controller-initiated  set-up,  which  consists  of  the  169  ms  as  above  and  the  260  ms  satellite  delay  of 
the  set-up  signal.  This  results  in  a  total  maximum  expected  delay  of  759  ms(70  +  260  + 169  +  260). 
The  earliest  time  after  the  PTT  that  a  controller  can  talk  without  any  loss  of  audio  (clipping  of 
words  ■  is  99  ms.  For  the  tests,  the  voice  delay  in  each  direction  between  controllers  and  pilots  is 
set  to  330ms  (70  +  260),  and  the  simulated  set-up  delay  is  99  ms. 

4.1 .5  The  following  tables  consolidate  the  above  information: 


Table  A1.  Real  World  Maximum  Delays  (MSECs) 


set-up 

voice 

’dead  ’  time 

TODAY 

pilot 

0 

0 

controller 

225 

0 

225 

VSCS 

pilot 

0 

70 

controller 

169 

70 

239 

TcDAY  +  SAT 

pilot 

0 

260 

controller 

485 

260 

745 

V-CS  +  SAT 

pilot 

0 

330 

controller 

169 

330 

759 

759 


Table  A2.  Test  Values  to  be  Used 


set-up 

voice 

’dead  ’  time 

TODAY 

pilot 

0 

0 

controller 

225* 

0 

225 

VSCS 

pilot 

0 

>0 

controller 

99* 

70 

239 

TODAY  +  SAT 

pilot 

0 

260 

controller 

225* 

260 

745 

VSCS  +  SAT 

pilot 

0 

330 

controller 

99* 

330 

759 

*  Minimum  delay  to  lose  no  controller  audio  (clip)  (audio  cue  to  controller’s  ear) 


4.2  Operational  Environment 

The  operational  environment  will  consist  of  the  observed  test  sectors,  the  support  sectors,  a  ghost 
sector  (not  under  test)  that  accept/issue  hand-offs  to  the  test  sectors,  and  the  simulation  facility, 
with  simulator  pilots  as  the  aircraft  and  the  communications  sources  to  the  controllers.  The  test 
configuration  will  consist  of  five  sectors  from  the  Atlanta  Center: 

•  Dublin  High,  sector  7 

•  Sinca  Arrival,  sector  8 

•  Macon  High,  sector  22 

•  South  Departure,  sector  21 

•  Clark  Hill  Ultrahigh,  sector  24 

where  the  first  three  sectors  are  the  observed  test  sectors  and  the  other  two  sectors  are  the  support 
sectors.  The  three  test  sectors  will  consist  of  two  high  altitude  sectors  and  one  low  altitude  sector 
with  one  high  altitude  sector  feeding  traffic  to  the  other  high  altitude  sector  and  the  second  high 
altitude  sector  feeding  traffic  to  the  low  altitude  sector.  All  sectors  will  be  under  operation  at  the 
same  time,  and  the  test  sectors  will  each  operate  at  different  load  levels  consistent  with  the  typical 
’busy  day’  loads  for  those  sectors,  as  defined  within  the  center.  The  sectors  are  considered  to  be 
representative  types,  and  the  particular  Atlanta  sectors  used  are  considered  by  the  Southern 
Regional  Office  representatives  as  good  examples  of  busy  sectors. 

4.3  Traffic  Load 

The  traffic  loads  to  be  used  during  the  tests  are  defined  as  medium,  high  and  high  density,  where 
medium  density  is  70%,  high  density  is  90%,  and  very  high  density  is  110%  of  the  peak  traffic 
density  as  measured  at  the  Atlanta  Center.  Each  observed  test  sector  will  be  subject  to  all  three 
traffic  loads  at  all  delays.  The  communications  loads  that  come  about  as  a  result  of  the  above 
traffic  loads  will  be  reviewed  during  the  shakedown  tests  and  the  scripts  will  be  adjusted  if 
necessary  to  assure  the  required  communications  traffic  loads,  up  to  the  specification  limits  of  the 
VSCS. 
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4.3.1. 

In  the  test  sectors,  each  simulator  pilot  will  control  only  a  single  aircraft  at  any  time,  to  ensure  no 
simulator  pilot  limitation  for  contention  on  the  voice  channels.  Up  to  10  pilots  per  controller  will 
be  provided  for,  and  exercised,  in  the  scripts. 

4.4  Scenarios 

Scenarios  have  been  prepared  for  the  three  traffic  loads  indicated  in  paragraph  3  and  will  be 
evaluated  during  the  shakedown  tests  with  changes  to  be  made,  if  needed  before  starting  the 
actual  tests.  In  addition,  some  changes  may  be  made  in  the  scripts  for  each  test  in  order  to 
eliminate  the  potential  for  controllers  and  pilots  to  become  too  familiar  with  the  script  and  reduce 
the  validity  of  the  tests. 

4.5  Human  Factors 

Three  types  of  personnel  will  be  involved  directly  in  the  performance  of  these  tests.  They  are  the 
simulator  pilots,  the  controllers,  and  the  observers,  with  the  following  duties  assigned  to  each 
group. 

4.5.1  Simulator  Pilots. 

There  will  be  approximately  37  simulator  pilots  available  for  the  tests  with  as  many  as  ten 
individual  pilots  in  the  same  sector  on  the  same  frequency  at  the  same  time,  thus  providing  the 
necessary  environment  for  communications  contention.  All  of  the  simulator  pilots  will  receive 
training  to  achieve  the  desired  communications  loading  effect  by  learning  how  and  when  to  access 
the  voice  channel  as  soon  as  it  appears  to  be  unused. 

4.5.2  Controllers 

Each  test  will  require  the  services  of  three  types  of  controllers.  There  will  be  a  test  subject  R 
controller  at  each  observed  test  sector,  who  will  be  a  radar  controller  experienced  in  operating 
the  selected  sector  at  Atlanta  ARTCC .  There  will  be  a  D  controller  at  each  observed  test  sector 
performing  the  D  position  functions,  who  will  be  an  experienced  Atlanta  ARTCC  data  controller, 
bt 1  will  not  be  a  test  subject.  There  will  be  an  R  controller  at  each  of  the  other  two  sector  positions, 
also  an  experienced  Atlanta  ARTCC  radar  controller,  who  will  not  be  a  test  subject  while  at  those 
positions.  Based  upon  the  qualifications  of  all  the  conti  oilers  involved  in  the  testing,  the  control¬ 
lers  will  be  at  different  sectors  and  will  perform  different  functions  during  the  set  of  tests  that  are 
run. 

4.5.3  Observers 

One  observer  will  be  required  for  each  of  the  three  test  sectors.  These  people  should  be  the  same 
for  all  of  the  tests.  They  will  be  experienced,  senior  radar  controllers,  ideally  Atlanta  controllers 
currently  serving  as  training  instructors,  and  will  observe,  monitor,  and  record  the  performance 
within  the  test  sector  assigned,  in  accordance  with  the  evaluation  process  in  the  test  plan.  The 
same  observers  should  be  used  during  the  entire  test  period.  They  will  also  be  serving  as  the  R 
controllers  during  the  shakedown  tests.  In  one  of  the  shakedown  tests,  they  will  perform  their 
duties  as  observers  to  further  verify  the  observer  procedures.  Observers  will  use  headsets  to  listen 
to  the  pilot  and  R  controller  audio  at  each  sector. 
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5.  Test  Bed  Development 

All  of  the  tests  will  be  conducted  at  the  FAA  Technical  Center  using  the  facilities  described  below. 

5.1  Test  Bed  Description 

The  test  bed  will  consist  of  the  following  elements: 

•  NAS  Enroute  Lab  with  the  Host  Computer,  the  9020E,  and  five  sectors  con¬ 
figured  as  part  of  the  Atlanta  Center. 

•  Digital  Simulation  Facility  with  up  to  37  simulator  operators  providing  aircraft 
targets  and  communications  with  the  controllers. 

•  Amecom  voice  switching  system  modified  with  adjustable  delay  circuits  to  pro¬ 
vide  the  desired  delay  times  and  communications  between  the  simulator 
operators  and  the  controllers.  The  set-up  delays  and  the  utterance  delays  will  be 
independently  controlled.  Capabilities  added  to  the  Amecom  are: 

-  0  to  1024  ms  voice  delay  in  1  ms  steps 

-  0  to  500ms  simulated  set-up  delay  by  delaying  the  effect  of  the  PTT  signal 
for  a/g  communication 

-  short  audio  cue  to  controller’s  ear  at  end  of  simulated  set-up  time,  cor¬ 
responding  to  time  after  which  no  audio  is  lost  (clipped). 

5.2  Test  Implementation 

Each  test  will  be  of  approximately  one  hour  duration  with  a  gradual  buildup  of  the  traffic  to  the 
desired  level.  The  desired  traffic  level  will  be  maintained  for  sufficient  time  to  collect  adequate 
data  for  analysis.  The  three  observed  test  sectors  will  be  staffed  with  both  an  R  and  D  controller, 
while  the  other  two  sectors  will  have  the  staffing  needed  to  support  the  operation  of  the  test 
sectors  .  An  observer  will  be  assigned  to  each  observed  test  sector  and  data  will  be  collected  for 
all  three  test  sectors  during  each  test  run.  There  will  be  three  test  runs  on  each  day  of  testing  with 
three  days  of  testing  per  week  for  a  total  of  nine  tests  a  week.  This  sequence  of  testing  will  be 
performed  for  three  weeks  with  a  new  group  of  Atlanta  controllers  being  the  test  subjects  each 
week. 

5.2.1 

Shakedown  testing  of  the  software,  scenarios,  pilots,  communication  hardware,  procedures,  etc. 
will  take  place  for  three  days  to  assure  that  all  aspects  of  the  testing  are  proper  and  can  go  ahead 
as  planned.  A  review  of  communication  loading  during  the  shakedown  tests  will  determine 
whether  the  scripts  need  modification.  Controllers  from  Atlanta  sectors  will  be  utilized  as  D 
controllers  for  the  observed  test  sectors  and  R  controllers  for  the  other  two  test  sectors;  the 
training  instructors  will  serve  as  the  R  controllers  for  the  three  observed  test  sectors.  In  addition, 
the  shakedown  tests  will  include  some  tests  with  the  training  instructors  performing  as  observers. 

5.3  Test  Procedure 

Table  A3,  depicts  the  experimental  conditions  to  be  experienced  by  three  ATCS  teams.  Each  team 
is  made  up  of  3  R  controllers  and  3  D  controllers.  These  6  individuals  will  operate  three  of  the 
five  sectors  provided  in  the  simulation;  the  remaining  two  sectors  will  be  manned,  and  will 
operate,  communicate  and  control  targets,  accept  handoffs,  and  give  handoffs,  but  will  not  be  test 
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sectors  nor  sectors  observed  by  the  test  observers.  A  ghost  communication  position  will  also 
handle  the  ground-ground  communications  outside  of  the  test  sectors. 

Communications  workload  levels  are  set  in  the  simulations  scenarios.  The  communications 
workload  levels,  driven  by  the  traffic  load  levels  in  the  scenarios,  are  considered  medium  (M), 
high  (H),  and  very  high  ( VH)  relative  to  the  Atlanta  ARTCC  operations  values. 

The  test  delay  conditions  as  shown  in  the  Table  A3  correspond  to  (0)  for  Baseline  (TODAY),  (1) 
for  VSCS  time  frame,  (2)  for  TODAY  with  Satellite,  and  (3)  for  the  VSCS  time-frame  with 
Satellite,  and  are  the  current  worst  case  delay,  and  the  three  other  delays,  which  correspond  to 
the  conditions  of  paragraphs  4.1.1, 4.1.2, 4.1.3,  and  4.1.4  respectively. 

Table  A4  shows  the  balanced  run  sequence  of  delays  combined  with  load  conditions  for  controller 
teams.  There  is  a  balance  of  load  conditions  over  controller  teams,  and  there  is  even  a  partial 
balance  over  sector  assignments  within  the  team.  Each  team  are  test  subjects  in  three  sector 
assignments  identified  as  a,  b,  and  c.  Each  team  also  experiences  the  baseline  delay  once  at  each 
sector.  This  layout  achieves  nearly  the  maximum  balance  for  three  teams  each  experiencing  9 
conditions.  With  few  exceptions,  there  is  no  load  condition,  delay  condition  or  sector  assignment 
that  is  immediately  repeated.  With  one  exception,  the  delay  and  load  conditions  are  well-dis¬ 
tributed  over  each  sequence  of  conditions.  In  order  to  satisfy  a  constraint  that  each  team  be  tested 
at  a  base-line  or  VSCS  delay  value  before  they  are  tested  at  the  longer  (satellite)  delay  values,  a 
small  adjustment  to  the  order  is  included,  and  potential  balance  weakening  is  acknowledged. 

This  design  balances  all  major  variables  and  partially  balances  the  minor  variables,  which  results 
in  an  experiment  with  a  minimum  of  ’confounding,’  and  a  maximum  of  ’analyzability.’ 

Each  R  controller  test  subject  will  participate  in  one  baseline  delay  test  at  all  three  traffic  density 
levels,  and  three  other  delay  tests  also  at  all  three  traffic  density  levels.  In  order  to  improve  the 
validity  of  the  tests,  controllers  will  be  rotated  through  all  three  test  sectors  and,  if  possible,  will 
not  be  a  test  subject  in  the  same  sector  on  consecutive  tests  .  See  Table  A5. 
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Table  A3.  Test  Assignments 


Team  and  Week 

Communication  Workload 

Sector 

Medium 

High 

Very  High 

Assignment 

0 

1 

2 

a 

1 

3 

0 

1 

b 

2 

3 

0 

c 

0 

2 

3 

a 

2 

1 

0 

2 

b 

3 

1 

0 

c 

2 

0 

3 

a 

3 

0 

3 

1 

b 

1 

2 

0 

c 

Table  A4.  Sequence  of  Test  Conditions 


Team 

Session  Num 

ber 

and 

Week 

1 

2 

3 

4 

5 

1  6 

7 

8 

9 

1 

1H 

1 

mm 

OH 

■ 

3H 

2M 

WBm 

a 

WBM 

ns 

b 

Hi 

c 

■MB 

c 

2 

1H 

2VH 

3M 

OH 

2H 

3VH 

C 

fmt. 

b 

C 

b 

Si 

a 

B. 

a 

3 

■ 

3VH 

2H 

' 

3H 

OVH  1 

2M 

1VH 

OH 

— 

a 

C 

WBm 

b 

c 

a 

b 

a 

Table  A5.  Assignment  of  Controllers  to  Sectors 


5.4  Test  Management 

A  Test  Manager  will  be  responsible  for  the  coordination  of  the  Technical  Center’s  preparation  of 
the  test  facilities,  test  scenarios,  hardware  and  software  changes  to  equipment,  schedules,  person¬ 
nel  availability,  and  for  the  content  of  the  Test  Plan  and  its  approval.  The  Test  Manager  will 
provide  for  the  Test  Directors  and  other  Technical  Center  personnel  to  assure  the  conducting  of 
the  tests. 

Test  Director(s)  from  the  Technical  Center,  under  the  direction  of  the  Test  Manager,  will  be 
responsible  for  conducting  the  tests.  They  will  assure  that  all  test  personnel  have  been  briefed, 
all  positions  are  staffed,  and  the  test  bed  is  fully  operational  before  the  start  of  each  test. 

Furthermore,  test  scripts,  questionaires,  observer  reporting  forms,  and  all  other  reporting  forms 
will  be  explained  and  distributed  prior  to  the  start  of  each  test  run.  Test  directors  will  provide  day 
to  day  supervision  of  the  activities  and  coordinate  the  starting,  running,  and  ending  of  the  tests,  as 
well  as  the  distribution  and  collection  of  questionaires,  and  the  collection  and  processing  of  the 
daily  digital  and  analog  tapes.  Representatives  of  the  sponsoring  organizations,  ATR,  AAP,  and 
ASM  will  assist  to  ensure  conformance  to  the  test  plan  and  to  maintain  an  appropriate  test 
environment. 

6.0  Test  Design/Description 

All  of  the  tests  have  the  same  basic  objective,  as  stated  in  paragraph  1.0,  to  determine  the  impact 
of  voice  transmission  delays,  within  a  realistic  operational  environment,  on  controller  workload 
and  performance.  Each  test  run  will  produce  specific  performance  data  from  two  primary  sources, 
controller  and  observer  answers  to  questionaires  after  each  run  and  observer  notes  and  records 
after  each  run.  Secondary  sources  of  performance  data,  recorded  digital  control  and  analog  voice 
recordings,  and  pilot’s  records,  will  provide  information  to  corroborate  and  complement  the 
primary  sources. 

6.1  Test  Type 

(to  be  provided) 

6.2  Objective  Of  Test 

(to  be  provided) 

6.3  Test  Description 

(to  be  provided) 

6.4  Questionaire  Development 

Two  questionaires  will  be  developed,  one  for  use  by  the  observers  and  one  for  use  by  the 
controllers.  The  observer  questionaire  will  record  subjective  and  objective  answers  to  questions 
and  include  observations  that  are  made  during  the  actual  running  of  the  tests.  The  controller 
questionaire  will  include  questions  about  peformance  and  perceived  performance  relative  to  the 
just-completed  test.  These  questionaires  will  be  used  initially  during  the  test  bed  shakedown,  with 
changes  made,  if  necessary,  prior  to  the  start  of  the  formal  tests. 
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6.4.1  Pilot’s  Records 

Pilots  will  keep  a  record  of  the  number  of  calls  they  each  make  to  a  controller  which  are  not 
answered. 

7.  Test  Evaluation 

7.1  Evaluation  Criteria 

(See  Addenda) 

7.2  Data  Reduction  and  Analysis 

(See  Addenda) 

7.3  Test  Result  Evaluation 

(See  Addenda) 

8.  Responsibilities 

The  organizational  responsibilities  for  each  area  of  the  testing  are  listed  below.  Specific  organiza¬ 
tional  units  and/or  persons  will  be  identified  in  the  test  procedures. 

8.1  Specification  of  Test  Parameters 

ATR  at  Headquarters  will  coordinate  with  AAP  and  ASM  and  provide  leadership  in  the  specifica¬ 
tion  of  test  requirements,  in  reviewing  test  plans  and  materials,  in  monitoring  test  conduct,  and  in 
reviewing  test  products. 

8.2  Development  of  Test  Bed 

ACN  will  be  responsible  for  the  set-up,  use  and  overall  development  of  the  simulated  Air  Traffic 
Control  test  bed  at  the  Technical  Center,  and  for  the  technical  correctness  and  performance  of 
the  test  bed. 

8.3  Test  Design/Description 

TSC  will  be  responsible  for  the  test  design/description  and  test  data  collection  documents  for  use 
by  the  test  subjects  and  observers,  in  support  of  AAP  and  ASM. 

8.4  Test  Management 

ACN  will  be  responsible  for  the  overall  test  management  at  the  FAA  Technical  Center. 

8.5  Test  Eva1  nation 

ACN/ATS/TSC  will  be  responsible  for  quantitative  and  qualitative  evaluation  of  the  tests,  with 
TSC  as  the  lead  organization. 

8.6  Report/Test  Plan  Preparation 

ACN  will  be  responsible  for  preparing  the  test  plan  and  for  the  preparation  of  the  report  of  the 
result  in  conjunction  with  TSC.  ATR  will  approve  the  test  plan. 
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8.7  Coordination 

ACN  will  coordinate  all  of  the  activities. 

9.  Resource  Requirements 

9.1  Funding 

AAP/ASM  will  provide  all  of  the  funding  necessary  to  accomplish  this  task. 

Cost  of  field  controllers,  etc  _ 

(Travel,  Per  Diem,  Salary)  _ 

Operation  of  Simulation  Facility _ 

Operation  of  the  ATC  Lab  _ 

All  project  people  _ 

Equipment,  supplies  _ 

Total  _ 

9.2  Staffing 

The  required  staffing  for  this  program  is: 

Shakedown  Tests  (one  week) 

Atlanta  training  instructors  3 
Atlanta  R  controllers  3  +  2  =  5 
Atlanta  D  controllers  3 
Ghost  position  1 
Simulator  pilots  26-30  est. 

Full  Tests  (each  week  for  three  weeks) 

Atlanta  training  instructors  3 

Atlanta  R  controllers  (sector  specific)  3 

Atlanta  R  controllers  2 

Atlanta  D  controllers  3 

Ghost  position  1 

Simulator  pilots  26-30  est. 

Test  management  and  support  12  est. 
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10. 


Schedule 


Test  Plan  Development  October  6,  1989 
Test  Bed  Shakedown  November  6-8, 1989 
Test  Schedule  November  28.29.30  daily  schedule  TBD 
Test  Report  (Draft)  January  19, 1990 
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Errata  and  Corrections 


1  0  (additions  for  information  and  clarification) 

4.2  (last  sentence  changed  to  clarify) 

Addenda 

7.1  Evaluation  Criteria 

Evaluation  will  be  based  upon  items  that  will  appear  on  the  observers’  computer  checklists  for 
use  during  test  runs,  questionaires  for  use  by  observers  after  each  run,  questionaires  for  use  by 
test  R-controllers  after  each  run,  as  well  as  the  limited  data  collected  by  the  simulation  pilots  and 
secondary  data  derived  from  Simulation  Analysis  Review,  digital  Amacom  controller  PTT,  and 
pilot  and  controller  voice  tapes.  In  general,  the  data  will  be  diagnostic  of  communications 
problems,  workload,  and  reduced  levels  of  performance  due  to  the  experimental  conditions. 
Computer  or  paper  (backup)  checklists  will  consist  of  tallies  for  step-ons,  requests  for  repeated 
pilot  transmissions,  conflict  alerts,  and  mistakes.  Observers’  questionaires  will  consist  of  items 
concerning  realism  of  the  simulation,  difficulty,  attention,  stress,  disruption  of  traffic  flow,  time 
spent  in  various  sub-tasks,  and  safety.  In  addition,  items  on  controllers’  questionaires  will  address 
simulated  air  traffic  load  and  complexity,  difficulty  in  completing  communications,  frustration, 
misunderstandings  in  communicating  with  simulation  pilots,  and  satisfaction  with  their  (the  test 
controllers’)  performance. 

7.2  Data  Reduction  and  Analysis 

Multiple  regression  techniques  will  be  employed  in  the  analysis  of  the  data.  It  is  anticipated  that 
dependent  variable  data  will  be  uploaded  into  the  SAS  statistical  analysis  system  and  analyzed 
using  the  GLM  (General  Linear  Model)  procedure.  This  will  provide  statistical  estimates  of  the 
contributions  of  delay  and  communications  load  to  the  obtained  results.  Data  reduction  and 
summary  statistics  from  all  tape  sources  will  be  provided  by  ACN  for  analysis  by  TSC.  Data 
reduction  from  all  other  sources  will  be  provided  by  TSC. 

7.3  Test  Resuit  Evaluation 

Where  appropriate,  tests  of  statistical  significance  will  be  used  to  assess  the  probability  that 
differences  in  results  due  to  delay  and  communications  load  are  due  to  chance. 
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Test  Plan  Approvals 

The  Test  plan  attached  satisfies  the  requirements  of  the  sponsoring  organizations  to  the  extent 
specified  in  the  document.  TBDs  will  be  separately  approved. 

_  for  ATR 

_  for  AAP 

_  for  ASM 

_  for  TSC 

_  for  ACN 
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Appendix  B 

Analysis  and  Determination  of  Communication  Delays 

for  Test  Purposes 

G.  Spanier 

This  analysis  includes  a  description,  in  textual  form,  of  the  actions  and  related  delays  that  exist  or 
will  exist.  They  are  followed  by  a  tabulation  of  the  delays,  for  additional  clarity.  This  is  followed 
by  an  explanation  of  the  basis  for  the  selection  of  the  specific  delay  quantities  that  are  (were)  used 
in  the  tests  in  order  to  simulate  the  delay  conditions.  The  actual  values  used  in  the  tests  produce 
EFFECTIVE  DELAYS  which,  as  far  as  the  human  user  of  the  system  is  concerned,  replicate  the 
real-time,  chronological  delays  of  an  actual  system. 

The  tests  incorporate  four  different  combinations  of  delays  that  simulate  delay  conditions  that  are 
expected  to  exist:  the  VSCS  time  frame  (abbreviated  to  ’VSCS’),  the  real  world  today  (indicated 
as  ’Today’),  and  both  with  and  without  satellite  links. 

Additional  discussion  of  delays  is  found  in  section  4.1,  DELAYS,  in  the  ATC  Voice  Communica¬ 
tions  Delay  Test  Plan,  10-17-89,  in  the  Appendix  of  this  report. 

The  criteria  for  the  communication  of  a  voice  message  in  the  context  of  this  analysis  are: 

a.  the  utterances  by  a  controller  are  not  clipped  or  missing  from  the  beginning  of  a 
message  because  the  transmitter  is  not  yet  ready  to  transmit  at  the  time  that  the 
utterance  reaches  it. 

b.  a  listener  on  the  frequency  hears  an  indication  such  as  lack  of  noise  level  or  audio 
which  indicates  that  the  frequency  is  NOT  in  use,  and  such  a  time  period  is 
called  ’dead  time.’ 

c.  the  controller  is  the  effective  ’controller’  of  the  frequency,  and  will  not  attempt  to 
communicate  (transmit)  until  a  pilot  who  is  talking  finishes,  and  the  controller 
actually  knows  this  by  message  context  and  noise  level. 

d.  Inadvertent  or  otherwise  malicious  transmission  by  a  pilot  while  the  frequency  is 
in  use  is  beyond  the  control  of  the  controller,  and  is  not  taken  into  account. 

e.  for  purposes  of  analysis,  the  shortest  times  for  waiting  to  transmit  after  a  channel 
is  available  are  counted,  i.e.  a  PTT  action  will  occur  immediately  after  a  fre¬ 
quency  is  free  to  use,  and  a  controller  is  considered  to  be  talking  as  soon  as  the 
transmitter  is  capable  of  transmitting. 

f.  the  starting  point  for  all  chronological  analysis  of  communications  is  the  end  of  the 
voice  transmission  by  a  pilot,  and  the  simultaneous  turn-off  of  his  transmitter. 

g.  a  squeal  is  usually  considered  a  positive  indication  to  a  listener  on  a  frequency 
that  two  or  more  transmitters  in  the  range  of  the  receiver  are  turned  on. 


A.  Description 

In  the  present  system,  the  delay  from  the  press  of  the  push-to-talk  (PTT)  switch  by  a  controller 
until  the  controller  receives  a  side  tone  noise  level  change  (audio  cue)  from  his  local  communica¬ 
tion  position  equipment  indicating  that  his  local  equipment  is  no  longer  in  the  receive  mode,  this 
delay,  is  just  a  few  milliseconds.  It  is  actually  the  local  electronics  responding  to  the  PTT  action, 
and  does  not  correspond  to  any  pertinent  event  in  time.  The  PTT  action  also  forces  the 
generation  of  a  signal  in  the  local  comm  equipment  which  is  called  ’transmitter  enable’  which, 
when  it  gets  to  the  transmitter,  will  instruct  the  transmitter  to  turn  itself  on  in  a  full-power  RF 
transmitting  mode.  The  transmitter  ’ON’  status  is  communicated  back  to  the  controller’s  position 
and  turns  on  an  indicator  light.  There  is  no  audible  signal  which  directly  corresponds  to  that 
indicator  light.  The  time  between  PTT  and  the  light  turning  on,  the  only  positive  indication  to  the 
controller  that  the  transmitter  is  in  the  transmission  mode,  is  423  milliseconds  (ms),  comprised  of 
194  ms  from  PTT  to  transmitter,  35  ms  for  transmitter  to  enable,  and  194  ms  for  the  return  signal 
to  reach  the  local  comm  equipment  to  turn  on  the  controller’s  light.  Waiting  for  the  light  to  turn 
on  before  speaking  is  not  and  will  not  be  normal  practice. 

However,  all  pilots  listening  on  the  frequency  know  that  someone  has  depressed  a  PTT  switch, 
since  they  would  immediately  hear  a  background  noise  level  change  from  their  receiver  due  to  a 
received  RF  signal  (audio  is  not  necessary),  but  they  do  not  know  who  has  initiated  the  com¬ 
munication.  If  it  is  a  controller  (the  important  case),  then  the  controller  would  have  initiated  the 
PTT  action  229  ms  (194  +  35)  earlier.  During  this  time  interval  a  communications  type  of 
interference  called  a  ’step-on’  (see  full  definition  elsewhere  in  this  report)  can  occur  if  a  PTT 
switch  is  depressed  by  a  pilot,  and,  as  a  result,  the  controller  will  not  be  aware  of  that  pilot’s  action, 
nor  of  the  step-on  condition,  and  the  pilot  will  not  be  aware  of  the  controller’s  action.  The  comm 
channels  are  used  in  a  simplex  mode;  there  is  no  received  audio  during  the  time  a  PTT  switch  is 
held  on. 

Pilot-initiated  communications  are  not  subject  to  any  delay  in  either  transmission  or  controller 
reception.  Step-ons  can  also  occur  if  pilots  simultaneously  depress  their  PTT  switches  or  are  not 
aware  of  another  pilot’s  transmission.  Depending  on  the  locations  of  the  aircraft  transmitting 
simultaneously,  others  listening  on  the  frequency,  including  the  controller,  may  hear  an  audio 
squeal  caused  by  two  RF  signals  interfering  with  and  affecting  the  normal  operation  of  the 
receiver’s  local  oscillator.  When  a  controller  hears  this,  it  is  a  clear  indication  that  all  or  a  portion 
of  a  pilot’s  communication  will  not  be  intelligible. 

For  the  case  of  the  use  of  a  ground-to-satellite-to-ground  link,  260  ms  (see  derivation  and 
references)  must  be  added  to  the  worst  case  Today  condition,  229  ms,  which  results  in  a  total  delay 
of  489  ms  before  a  pilot  is  aware  that  the  frequency  is  in  use. 

For  the  VSCS  time  frame  system,  the  delays  have  different  sources,  and  are  identified  as  follows: 
The  PTT  to  VSCS  output  is  30  ms,  the  local  RCE  to  transmission  equipment  is  50  ms,  the 
transmission  equipment  is  4  ms,  the  transmission  equipment  to  remote  RCE  output  is  50  ms,  and 
the  transmitter  enable  is  35  ms  for  a  total  of  169  ms,  with  an  additional  134  ms  for  the  controller 
to  receive  the  transmitter  ’on’  status  indication. 
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Once  the  controller  starts  to  talk,  there  can  be  up  to  a  70  ms  utterance  delay  from  local  comm 
equipment,  from  microphone  to  aircraft  speaker.  This  delay  is  also  present  in  communications 
from  pilot’s  microphone  to  controller’s  speaker.  The  corresponding  delay  time  interval,  for  a 
potential  step-on  of  a  controller,  is  169  ms  from  transmitter  set-up  plus  70  ms,  or  239  ms,  after 
set-up  for  the  voice  utterance  to  get  to  the  transmitter.  The  time  for  receipt  of  the  controller’s 
first  utterance  is  373  ms  if  he  waits  for  the  transmit  enable  light  -  waiting  for  the  light  to  turn  on 
is  not  the  standard  or  normal  process  today,  nor  will  it  be  in  the  future. 

When  a  pilot  completes  his  transmission,  there  is  no  delay  before  other  pilots  know  this,  but  there 
is  that  70  ms  delay  before  the  controller  knows.  This  delay  must  be  added  to  the  time  that  another 
pilot  is  not  aware  that  anyone  else  is  transmitting,  as  explained  in  the  previous  paragraph,  which 
changes  the  pertinent  value  to  239  ms  (169  +  70). 

The  other  delay  that  can  exist  is  the  earth-to-satellite-to-earth  link  delay,  which  is  a  approximately 
260  ms  (see  derivation  and  references).  This  value  should  be  added  to  the  worst  case  VSCS 
condition  (239  ms),  which  in  that  case  results  in  a  total  delay  of  499  ms  before  a  pilot  is  aware  that 
the  frequency  is  in  use. 


B.  Tabulation 

The  following  tabulation  of  voice  communication  delays  in  Today’s  system,  the  VSCS  system,  and 
with  and  without  satellite  link,  is  based  on  numbers  provided  by  the  SEIC  to  the  VSCS  program, 
by  specification  values,  and  confirmed  by  the  program  personnel,  and  uses  worst  case,  99.9%ile 
response  times,  and  other  values  based  on  measured  and  specification  values  and  engineering 
practice.  The  figures  and  analysis  are  specifically  directed  towards  enroute  sector  air/ground 
communications. 


1.  Set  Up  Time 

Real  World  Today 


70 

ms 

60 

ms 

4 

ms 

60 

ms 

35 

ms 

229 

ms 

from  PTT  to  output  of  4-channel  AJR.TCC  comm  equipment 
from  comm  equip  output  to  tone  control  equip  output 
transmission  equipment  input  to  tone  control  input 
from  tone  control  equipment  input  to  xmitter  input 
xmitter  turn-on  to  full  power 

total  setup  time  (line  is  enabled  for  controller  speech) 
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Return  path  controller 

60  ms  xmitter  enable  to  output  of  tone  control  equip 

4  ms  transmission  equip  input  to  output 

60  ms  tone  control  input  to  comm  equip  input 

70  ms  comm  equip  input  to  side-tone  cue  to  controller’s  ear 

194  ms  total  enable  cue  return  time  (controller  knows  xmitter  is  enabled) 

♦♦♦Controllers  do  not  normally  wait  for  any  transmitter  ’on’  signal,  either 
audible  or  visual,  before  talking.*** 


VSCS 

30  ms  from  PTT  to  output  of  VSCS 
50  ms  from  VSCS  output  to  RCE  output 
4  ms  from  RCE  output  to  TE  out 
50  ms  from  TE  output  to  RCE  output 
35  ms  from  xmitter  input  to  full  power  output 
169  ms  total  setup  time 


From  xmitter  enable  back  to  controller’s  ear 

50  ms  xmitter  enable  to  RCE  output 
4  ms  from  RCE  output  to  TE  output 
50  ms  from  TE  output  to  RCE  output 
30  ms  from  VSCS  input  to  side-tone  cue  to  controller’s  ear 
134  ms  total  enable  cue  return  time 

♦♦♦Controllers  do  not  normally  wait  for  any  transmitter  ’on’  signal,  either 
audible  or  visual,  before  talking.*** 
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2.  Voice  Propagation  Delay  -  Controller/Pilot 

Real  world  today 

0.0  ms  from  microphone  input  to  speaker  output 

VSCS  Time  Frame 

70  ms  maximum  from  microphone  to  speaker 

3.  Satellite  Link  Delay 

For  a  synchronous  satellite  orbiting  the  earth  at  22,400  NM,  with  no  significant  (less  than  1.0  ms) 
delay  of  retransmission  in  the  satellite,  the  propagation  delay  due  to  distance  in  air  from  the  earth 
station  to  the  satellite  and  then  back  to  an  earth  station  is  nominally  262.927  ms.  260  ms  is  used 
in  this  analysis.  (See  derivation) 

Set-up  delay  with  satellite  -  Today 

229  ms  set-up  delay  w/o  satellite 

260  ms  transmitter  enable  signal  delay  through  satellite 

489  ms  total 
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vscs 


70  ms  delay  before  controller  is  aware  pilot  has  completed  his  transmission 
169  ms  set-up  delay  w/o  satellite 
260  ms  transmitter  enable  signal  delay  through  satellite 

489  ms  total 

4.  Delay  Overlap 

For  the  Today  system,  both  transmitter  control  signal  and  audio  travel  on  the  same  path  to  the 
transmitter,  which  means  that  the  actual  delay  between  a  controller  PTT  action  and  starting  to 
speak  must  take  into  account  the  229  ms  delay  before  the  transmitter  is  ready  to  transmit  (to 
prevent  the  first  part  of  an  utterance  from  being  clipped).  Conversely,  if  a  controller  speaks  at 
the  same  time  that  he  presses  the  PTT  switch,  approximately  229  ms  of  audio  at  the  start  of  a 
message  will  not  be  transmitted  (the  transmitter  may  be  transmitting  a  few  milliseconds  before  it 
gets  up  to  full  power  output).  This  applies  at  all  times  of  start  of  transmission. 

For  the  VSCS  system,  the  transmitter  enable  control  signal  travels  through  the  local  comm 
equipment  on  one  path  with  delay  independent  of  the  path  and  the  delay  that  audio  experiences 
passing  through  the  local  comm  equipment.  The  effect  is  that  in  order  to  avoid  clipping  audio,  a 
controller  need  only  wait  169  ms  (the  set-up  time)  minus  70  ms  (the  audio  delay  time),  or  only  99 
ms  after  PTT  before  speaking.  With  this  amount  of  delay,  the  audio  will  arrive  at  the  transmitter 
at  just  the  exact  time  that  the  transmitter  will  be  up  to  transmitting  power.  Conversely,  if  a 
controller  speaks  at  the  same  time  as  PTT,  only  99  ms  of  audio  will  be  lost.  This  number  is  a  worst 
case  number,  greatly  affected  by  local  communication  system  voice  loading. 

This  delay  overlap  requires  a  careful  set-up  during  the  testing,  to  insure  that  effective  delay  is 
utilized  in  the  simulation  laboratory. 

5.  Example 

Example  of  critical  communication  problem,  with  all  delays  shown  in  proper  time  relationship: 
Pilot  PI  is  talking. 

Pilot  P2  wants  to  call,  and  is  waiting  for  PI  to  finish. 

Controller  C  listens  to  ?1,  and  then  want-  to  talk  to  a  Pilot  P3  (or  even  to  PI  again). 
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TODAY 


VSCS 


PI  talks 
CPTT 


audio 


h29 1 


dead  time 


c 

J  audio 

P2  PTT 

P3  hears 

squeal 

PI  hears 

squeal 

PI  talks  audio 


C  hears  70 


C 


PTT 


C  talks 


99 


audio 


P3  hears 


70 


audio 


P2  would 
talk  if 
C  didn't 


audio 


C  would 
hear  P2 
if  C  didn't 
PTT 


70 


audio 


P2PTT 


P3  hears 
PI  hears 


sqaeal 


squeal 


The  potential  for  step-ons  at  the  end  of  a  pilot’s  communication  is  directly  related  to  the 
opportunity  for  step-ons,  the  effective  dead  time. 

All  pilots  hear  the  controller  finish  an  utterance  at  the  same  time,  today  and  VSCS,  so  there  is  no 
cWay  difference  effect. 
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C.  Delay  Values  Chosen 


Real  World  Maximum  Delays 


Set  -up 

Voice 

Dead  Time 

Today 

pilot  0  ms 

ATCS  229  ms 

0  ms 

0  ms 

0  +  0  +  229  +  0  =  229  ms 

VSCS 

pilot  0  ms 

ATCS  169  ms 

70  ms 

70  ms 

0  +  70  +  169  +  70-70  =  239  ms 

Today/Sat 

pilot  0  ms 

ATCS  229  +  260  ms 

260  ms 

260  ms 

260  +  229  +  260  =  749  ms 

VSCS/Sat 

pilot  0  ms 

ATCS  169  +  260ms 

70  +  260  ms 

70  +  260  ms 

330  +  169  +  330-70  =  759  ms 

In  order  to  allocate  the  proper  delay  values  to  the  simulation  equipment,  to  produce  the  effective 
delays  experienced,  the  test  values  are  chosen  as: 


Simulation  Test  Values 


Delay  Before 

No  Clipping 
of  Audio 

Audio 

Delay 

Effective  Dead  Time 

Today 

pilot  0  ms 

ATCS  225  ms* 

0  ms 

0  ms 

0  +  0  +  229  +  0  =  225  ms 

VSCS 

pilot  0  ms 

ATCS  99  ms 

70  ms 

70  ms 

70  +  99  +  70  =  239  ms 

Today/Sat 

pilot  0  ms 

ATCS  225  ms* 

260  ms 

260  ms 

260  +  225  +  260  =  745  ms 

VSCS/Sat 

pilot  0  ms 

ATCS  99  ms 

330  ms 

330  ms 

339  +  99  +  330  =  759  ms 

*  Rounded  to  225  to  indicate  variability  of  this  measured  value. 
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Appendix  C 

Simulation  Validity 

Checks  on  the  validity  of  the  simulation  consisted  of  automatically  recorded  data  and  question¬ 
naire  items.  Automatic  counts  were  made  of  air  traffic  density  and  communication  activity.  The 
latter  was  indicated  by  the  frequency  with  which  the  R-controllers  keyed  their  microphones. 
Tables  1  and  2  show  the  traffic  density  and  communication  activity  achieved  in  the  simulation. 
The  intended  air  traffic  density  of  the  simulated  sectors  was  presented  in  Section  2.4.2.  The  traffic 
density  values  from  the  simulation  show  good  agreement  with  the  intended  traffic  levels,  the 
greatest  discrepancy  being  only  1.49  aircraft  less  than  intended  in  the  Macon  High  high  com¬ 
munications  load  condition. 

Three  R-controller  questionnaire  items  addressed  the  realism  of  the  simulation  pilots’  com¬ 
munications  and  simulated  air  traffic.  An  item  concerning  the  realism  of  the  air  traffic  in  the 
simulation  showed  a  mean  rating  of  3.30  on  a  four-point  scale,  where  1.0  was  “many  major 
differences,”  2.0  was  “some  major  differences,”  3.0  was  “some  minor  differences,”  and  4.0  was 
“almost  no  differences.”  Items  concerning  the  R-controllers’  subjective  impressions  of  simulation 
pilot  realism  indicated  that  the  simulation  pilots  were  perceived  to  have  made  as  many  requests 
of  the  controllers  as  real  pilots,  as  indicated  by  a  mean  rating  of  3.32  on  a  five-point  scale,  where 
1.0  was  “many  fewer,”  3.0  was  “the  same,”  and  5.0  was  “many  more."  They  were  also  perceived 
to  speak  at  the  same  rate  as  real  pilots.  Here  the  mean  was  again  3.32  on  a  five-point  scale,  where 
1.0  was  “much  more  slowly,”  3.0  was  “no  difference,”  and  5.0  was  “much  more  rapidly.” 
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Appendix  D 

Questionnaires 

This  appendix  contains  the  questionnaires  as  well  as  all  the  data  collected  from  the  question¬ 
naires  given  to  both  the  operations  observers  and  to  the  R-controllers.  The  first  four  columns 
of  both  tables  present  a  row  number  (one  for  each  respondent),  then  the  level  of  load  (Id), 
delay  (dy),  and  sector  (sc),  in  that  order.  The  remaining  columns  are  labelled  with  the  question 
number  from  the  questionnaire.  The  letter  "y"  is  used  to  represent  missing  data  for  a  question 
with  numeric  responses  and  the  letter  "x"  is  used  to  represent  missing  data  for  a  question  with 
character  responses. 
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Study  of  the  Effect  of  Equipment  Delays  on  Simulated  Air  Traffic  Operations 
Tracking  Code _  Date _  Time _  Sector _ 


Post-run  Questionaire  for  Test  Radar  Controllers 

The  following  items  pertain  to  the  BUSIEST  POINT  of  the  test  run  that  you  just 
completed.  Your  answers  will  be  kept  confidential. 


1.  At  its  busiest  point,  how  would  you  rate  the  traffic  load  as  compared  to  actual  peak 
traffic  for  this  sector? 

a.  50%  of  actual  traffic 

b.  60%  of  actual  traffic 

c.  70%  of  actual  traffic 

d.  80%  of  actual  traffic 

e.  90%  of  actual  traffic 

f.  100%  of  actual  traffic 

g.  110%  of  actual  traffic 

h.  120%  of  actual  traffic 

i.  130%  of  actual  traffic 

j.  140%  of  actual  traffic 

k.  1 50%  of  actual  traffic 


2.  How  realistic  was  the  simulated  air  traffic  compared  to  actual  traffic  for  this  sector? 

a.  almost  no  difference  between  simulation  and  real  air  traffic 

b.  some  minor  differences 

c.  some  major  differences 

d.  many  major  differences  between  simulation  and  real  air  traffic 


2a.  If  you  answered  c  or  d  in  item  1,  did  these  differences  affect  your  work? 

a.  yes,  they  made  it  easier. 

b.  yes,  they  made  it  more  difficult. 

c.  no,  the  differences  did  not  affect  the  difficulty  of  my  work. 


2b.  Please  describe  the  difference(s): 
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3.  Please  compare  the  performance  of  the  simulation  pilots  with  that  of  actual  pilots: 


3a.  Actual  pilots  speak: 


1  2 

much  more  slowly 

3 

no  difference 

4 

5 

much  more  rapidly 

3b.  Actual  pilots  make: 

1  2 
many  fewer 
requests 

3c.  Actual  pilots  make: 

3 

the  same  number 
of  requests 

4 

5 

many  more  requests 

a.  the  same  types  of  requests 

b.  different  types  of  requests 

If  you  answered  b,  please  list  some  examples: 


3d.  Please  describe  any  other  differences  between  the  performance  of  the  simulation 
pilots  and  real  pilots  that  may  have  affected  your  work: 


3e.  Indicate  any  difficulties  you  experienced  in  communicating  with  simulation  pilots: 

1  2  3  4  5 

no  some  many 

difficulties  difficulties  difficulties 

3f.  If  any  difficulties  occurred,  please  describe  the  nature  of  the  difficulties. 


4.  How  does  the  intercom  system  used  in  this  test  run  compare  to  the  communications 
equipment  that  you  are  used  to?  You  may  skip  this  item  after  having  answered  it 
after  the  first  run. 

4a.  Voice  Clarity:  The  intercom  system  was 

1  2  3  4  5 

much  clearer  the  same  much  less  clear 
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4b.  Amount  of  Static:  The  intercom  system  has 


1  2  3  4  5 

much  less  static  the  same  much  more  static 


4c.  Amount  of  Crosstalk:  The  intercom  system  has 

1  2  3  4  5 

much  less  the  same  much  more 

crosstalk  crosstalk 


5.  Please  describe  any  problems  with  the  equipment  used  in  the  preceeding  test  run: 


6.  Indicate  how  the  D-Controller  affected  your  workload. 

1  2  3  4  5 

greatly  decreased  my  workload  greatly  increased 

my  workload  not  affected  my  workload 

7.  If  you  have  been  assigned  to  this  simulated  sector  before,  in  this  study,  please  rate  the 
similarity  of  the  scenario  (flights,  flight  plans,  traffic  patterns,  etc.)  to  the  earlier  run(s). 

1  2  3  4  5 

completely  some  identical 

different  similarity 

7a.  If  there  was  some  similarity  to  that  run  (answers  2,3,4  or  5),  what  was  similar? 
Please  describe  any  similarities  in  the  space  below: 


8.  The  amount  and  complexity  of  traffic  in  the  simulation  was: 

1  2  3  4  5 

very  low  moderate  very  high 

9.  Indicate  the  amount  of  attention  or  concentration  required  to  control  the  simulated 
air  traffic: 

a.  little:  my  full  attention  was  needed  only  part  of  the  time. 

b.  moderate:  my  full  attention  was  needed  most  of  the  time. 

c.  high:  my  full  attention  was  needed  all  of  the  time. 

d.  too  high:  impossible  to  pay  sufficient  attention  to  some  aspects  of  the  task. 
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10.  Rate  the  cause(s)  of  the  scenarios’  difficulty  (even  if  it  was  easy  and  required  little 
attention)  by  placing  a  number  from  0-9  in  each  blank,  where  9  indicates  a  very 
important  cause  and  0  indicates  no  importance. 

a.  Number  of  flights: _ . 

b.  Number  of  voice  communications: _ . 

c.  Complexity  of  air  traffic: _ . 

d.  Simulation  pilots’  responsiveness  to  your  calls: _ . 

e.  Call  blocking,  hesitations,  step-ons: _ . 

f.  Other  communications  problems: _ . 

g.  Nonroutine  events: _ . 

h.  Other: _ . 

10a.  If  you  chose  a  number  other  than  zero  for  h,  please  describe  the  other  cause  of 
difficulty: 


11.  Please  rate  the  overall  level  of  stress  that  you  experienced. 

1  2  3  4  5 

no  stress  moderate  excessive 

stress  stress 

12.  Were  you  satisfied  with  your  performance  on  this  test  run? 

a.  very  satisfied 

b.  somewhat  satisfied 

c.  somewhat  dissatisfied 

d.  very  dissatisfied 

12a.  If  you  were  dissatisfied  (c  or  d),  why  was  this  the  case? 

Place  a  number  from  0-9  in  each  blank,  where  9  indicates  a  very  important  reason 
and  0  indicates  no  importance. 

a.  Number  of  flights: _ . 

b.  Number  of  voice  communications: _ . 

c.  Complexity  of  air  traffic: _ . 

d  Simulation  pilots’  responsiveness  to  your  calls: _ . 

e.  Call  blocking,  hesitations,  step-ons: _ . 

f.  Other  communications  problems: _ . 

g.  Nonroutine  events: _ . 

h.  Other: 


Please  use  the  space  below  for  any  other  comments: 
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Study  of  the  Effect  of  Equipment  Delays  on  Simulated  Air  Traffic  Operations 
Tracking  Number _ Date _ Time _ Sector _ 


CONTROLLER  OBSERVER’S  END  OF  RUN  OBSERVATIONS 

All  of  the  following  items  pertain  to  the  effects  of  experimental  conditions  upon 
participating  air  traffic  controllers.  These  conditions  are  arranged  to  test  the  impact  of 
new  equipment  by  finding  the  conditions  under  which  a  lower  standard  of  performance 
occurs.  Therefore,  a  lower  standard  of  performance  due  to  these  experimental 
conditions  is  anticipated.  No  conclusions  regarding  the  proficiency  of  these  research 
participants  are  warrented  by  their  performance  during  these  tests.  Any  use  of  the  data 
or  conclusions  of  this  study  as  indications  of  the  operational  performance  of  the 
participating  controllers  would  constitute  a  misinterpretation  of  the  data  and  the 
conclusions  of  the  study. 

Please  describe  any  uncorrected  mistakes  that  you  listed  during  the  preceeding  test  run. 
It  is  assumed  that  any  and  all  of  these  mistakes  are  due  to  the  experimental  conditions 
imposed  upon  the  controller,  and  are  not  indicators  of  controllers’  operational 
performance. 

Uncorrected  Mistakes  (Mis-Speaking): 


The  following  items  pertain  to  the  BUSIEST  POINT  of  the  test  run  that  you  just 
observed. 

1.  Please  compare  the  performance  of  the  simulation  pilots  at  the  busiest  point  in  the 
test  run  with  that  of  real  pilots: 

la.  Actual  pilots  speak: 

a.  more  slowly 

b.  more  rapidly 

c.  no  difference 

lb.  Actual  pilots  make: 

a.  fewer  requests 

b.  more  requests 

c.  as  many  requests 

lc.  Actual  pilots  make: 

a.  the  same  types  of  requests 

b.  different  types  of  requests 

If  you  answered  b,  please  list  some  examples: 

ld.  Please  describe  any  other  differences  between  the  performance  of  the  simulation 
pilots  and  real  pilots  that  may  have  affected  the  controller’s  work: 
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2.  Were  all  of  the  simulation  pilots’  initial  calls  answered? 


1  2 
all  were  answered 


3  4  5 

some  were  not  many  were  not 

answered  answered 


3.  About  how  long  did  the  controller  spend  at  various  activities? 


a.  Talking  with  pilots:  % 

b.  Talking  with  D-controller:  % 

c.  Writing  on  flight  strips:  % 

d.  Ground  communications:  % 

e.  No  observable  activity:  _ % 

f.  Other:  _ % 


=  100% 

3a.  If  you  placed  a  number  other  than  zero  in  f,  please  describe  the  other  activity: 


4.  Indicate  how  the  D-Controller  affected  the  R-Controller’s  workload. 

1  2  3  4  5 

greatly  decreased  R’s  workload  greatly  increased 

R’s  workload  not  affected  R’s  workload 


5.  Please  rate  the  overall  level  of  the  controller’s  stress  as  indicated  by  any  signs  of 
stress  that  you  noticed. 

1  2  3  4  5 

no  stress  moderate  excessive 

stress  stress 

6.  Indicate  the  amount  of  attention  or  concentration  required  to  control  the  simulated 
air  traffic: 

a.  little:  controller’s  full  attention  was  needed  only  part  of  the  time. 

b.  moderate:  controller’s  full  attention  was  needed  most  of  the  time. 

c.  high:  controller’s  attention  was  needed  all  of  the  time. 

d.  too  high:  impossible  to  pay  sufficient  attention  to  some  aspects  of  the  task. 

7.  Rate  the  cause(s)  of  the  scenarios’  difficulty  (even  if  it  was  easy  and  required  little 
attention)  by  placing  a  number  from  0-9  in  each  blank,  where  9  indicates  a  very 
important  cause  and  0  indicates  no  importance. 

a.  Number  of  flights: _ . 

b.  Number  of  voice  communications: _ . 

c.  Complexity  of  air  traffic: _ . 

d.  Simulation  pilots’  responsiveness  to  calls: _ . 

e.  Call  blocking,  hesitations,  step-ons: _ . 

f.  Other  communications  problems: _ . 

g.  Nonroutine  events: _ . 

h.  Other: 
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7a.  If  you  chose  a  number  other  than  zero  for  h,  please  describe  the  other  cause  of 
difficulty: 


8.  Did  the  experimental  conditions  cause  the  controller  to  overcontrol  the  air  traffic? 

a.  No  overcontrol  was  apparent. 

b.  Some  overcontrol  was  apparent. 

c.  Extensive  overcontrol  was  apparent. 

9.  How  smooth  was  the  flow  of  simulated  air  traffic? 

a.  not  disrupted 

b.  moderately  disrupted 

c.  severely  disrupted 

10.  Please  discuss  anything  that  you  observed  in  this  run  that  might  affect  safety  in 
actual  operations: 


Please  use  the  space  below  for  any  other  comments. 
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Appendix  E 

Analysis  Procedures 

In  order  to  develop  unbiased  and  accurate  comparisons  between  the  various  categories  of  delay 
it  was  desired  that  the  sessions  incorporate  each  level  of  delay  with  other  independent  variables 
(experimental  conditions)  in  a  balanced  manner,  i.e.,  if  delay  1  appears  X  percent  of  the  time 
under  medium  communication  workload  then  delay  2  also  appears  X  percent  of  the  time  under 
medium  communication  workload.  The  experimental  design  achieved  a  maximum  degree  of 
balance  for  the  size  of  the  experiment,  but  perfect  balance  was  not  achieved.  Even  with  an 
imperfectly  balanced  design  the  analysis  can,  to  an  extent,  correct  for  imbalance.  This  assumes 
that  the  design  is  analyzable  -  roughly  that  there  is  a  representative  mixture  of  combinations  of 
conditions  in  the  design.  Although  the  design  was  not  completely  balanced,  it  was  analyzable,  and 
corrections  for  lack  of  balance  were  made. 

Before  describing  the  regression  model  that  was  used,  it  is  necessary  to  consider  the  data  set  on 
which  it  operated.  The  regression  was  performed  using  the  Statistical  Analysis  System  (SAS). 
The  data  set  consisted  of  81  records;  27  experimental  trials  (or  sessions)  and  three  test  sectors  in 
each  session.  Each  record  included  data  on  a  number  of  variables  including  communication 
workload,  delay,  and  subject.  The  communication  workload  and  delay  values  were  the  same  for 
all  three  records  pertaining  to  a  given  session  (i.e.  the  records  from  the  three  test  sectors).  The 
subject  variable  indicated  which  of  the  R-controllers  worked  in  the  session/sector  combination 
that  created  that  record.  In  addition  to  the  four  independent  variables,  each  of  the  81  records 
included  several  dependent  variables.  These  were  counts  of  various  events  which  occurred  during 
the  session.  For  example,  one  was  the  count  of  controller-pilot  step-ons. 

The  regression  model  that  was  used  (Model  1  in  Appendix  G)  specified  the  above  four  inde¬ 
pendent  variables  and  also  the  interactions  of  delay  with  communication  workload  and  sector. 
The  SAS  GLM  (General  Linear  Model)  program  partitions  the  sum  of  the  squares  in  each 
dependent  variable  among  the  specified  factors  and  interactions  and  uses  the  residual  sum  of  the 
squares  to  estimate  a  mean  square  error  to  form  a  basis  for  deciding  statistical  significance  for  all 
comparisons  to  be  made  in  the  analysis.  Each  factor,  interaction,  and  even  comparisons  of 
specific  levels  of  the  independent  variables  is  tested  for  significance  using  the  mean  square  error. 
The  comparisons  of  various  conditions  may  be  made  using  the  least  square  (LS)  means  generated 
by  SAS.  The  LS  means  are  estimates  of  the  average  values  of  dependent  variables  under  various 
conditions  specified  by  combinations  of  independent  variables.  These  estimates  are  corrected  for 
unbalance  in  the  experimental  design.  For  example,  LS  means  for  step-ons  were  computed  at 
each  level  of  the  delay.  These  estimate  the  mean  number  of  step-ons  at  each  level  of  delay  and 
are  corrected  for  the  fact  that  a  given  delay  was  not  necessarily  tested  equally  often  under  each 
communication  workload  condition  nor  with  each  subject. 

The  output  of  the  SAS  GLM  procedure  gives  information  for  deciding  on  the  statistical  sig¬ 
nificance  of  the  individual  factors  and  interactions  based  on  their  “partial”  sums-of-squares.  The 
LS  means  can  also  be  obtained  as  well  as  “p-diffs”.  The  latter  are  for  assessing  the  significance  of 
differences  in  LS  means.  The  SAS  GLM  output  also  includes  an  R2  value  and  a  probability  value 
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for  the  model.  The  R2  value  indicates  the  extent  to  which  the  dependent  variable  is  affected  by 
all  of  the  independent  variables  in  the  regression.  The  model  probability  value  gives  information 
for  determining  the  significance  of  the  independent  variables  considered  together. 

When  the  LS  means  of  all  levels  of  a  factor  are  to  be  compared  then  a  correction  for  multiple 
testing  is  required.  In  SAS,  means  and  simple  factors  (e.g.,  differences  in  step-ons  due  to  different 
categories  of  delay)  are  corrected,  but  not  compound  factors  (e.g.,  interactions).  The  correction 
of  interactions  requires  a  separate  analysis,  using  Tukey’s  HSD  (Honestly  Significant  Difference). 
Consequently,  compound  factors  are  constructed  first  and  then  the  regression  (GLM)  for  con¬ 
structing  the  Tukey  HSDs  is  run.  This  regression  uses  essentially  the  same  model  but  the  variables 
are  coded  differently  (using  the  compound  variables).  Tukey’s  HSDs  permit  significance  state¬ 
ments  which  are  controlled  to  limit  the  total  Type  1  Error  even  if  all  possible  differences  are  of 
potential  interest.  Without  the  Tukey  HSD,  there  would  be  a  possibility  that  some  effects  noted 
in  the  study  which  might  appear  to  be  statistically  significant  were  really  not.  Since  the  HSD 
analysis  corrects  for  multiple  testing  we  use  a  significance  level  of  .05  for  it.  However,  in  general 
we  attempt  to  limit  the  cumulative  error  using  .01  as  the  level  of  significance. 

All  data  from  the  first  session  was  lost  due  to  a  procedural  error  in  the  simulation.  Also, 
automatically  recorded  data  from  three  sessions  was  lost  due  to  problems  with  magnetic  tape 
storage.  The  effect  of  the  missing  data  was  negligible;  sufficient  data  was  obtained  for  the 
estimation  of  the  parameters  needed  to  make  all  of  the  planned  comparisons. 
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Appendix  F 

SAS  Output 


This  appendix  contains  the  SAS  output  from  the  regressions  used  for  the  study.  The  dependent 
variables  analyzed  are:  Corrected  Mistakes  (CM),  Uncorrected  Mistakes  (UM),  Step-ons 
recorded  in  the  controller  lab  (S0  +  S1 +S2),  Step-ons  recorded  in  the  pilot  lab  (STEPS),  Pilot- 
Pilot  Step-ons  (PP),  Stress  reported  by  the  operations  observers,  Stress  reported  by  the  control¬ 
lers,  Attention  and  Concentration  reported  by  the  operations  observers,  Attention  and  Con¬ 
centration  reported  by  the  controllers. 
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CLASS  LEVEL  INFORMATION 


CLASS 

LEVELS 

VALUES 

SUBJ 

9 

1  2 

3 

4  5  6  7  8  9 

LOAD 

3 

1  2 

3 

DELAY 

4 

0  1 

2 

3 

SECTOR 

3 

1  2 

3 

NUMBER 

OF  OBSERVATIONS 

IN 

DATA  SET  =81 

DEPENDENT  VARIABLE:  CM 


SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

27 

547.17611374 

20.26578199 

ERROR 

53 

209.51524428 

3.95311782 

CORRECTED  TOTAL 

80 

756.69135802 

MODEL  F  = 

5.13 

PR  >  F  =  0.0001 

R-SQUARE 

C.V. 

ROOT  MSE 

CM  MEAN 

0.723117 

64.9386 

1.98824491 

3.06172840 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

201.53580247 

6.37 

0.0001 

LOAD 

2 

164.31142498 

20.78 

0.0001 

DELAY 

3 

2 . 10907822 

0.18 

0.9110 

SECTOR 

2 

157.74235917 

19.95 

0.0001 

DELAY*SECTOR 

6 

10.53710641 

0.44 

0.8458 

LOAD* DELAY 

6 

10.94034250 

0.46 

0.8338 
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SOURCE 

DP 

TYPE  III  SS 

P  VALUE 

PR  >  F 

SUBJ 

8 

171.30618429 

5.42 

0.0001 

LOAD 

2 

150.25528223 

19.00 

0.0001 

DELAY 

3 

1.37785309 

0.12 

0.9502 

SECTOR 

2 

158.31943825 

20.02 

0.0001 

DELAY* SECTOR 

6 

10.53710641 

0.44 

0.8458 

LOAD*DELAY 

6 

10.94034250 

0.46 

0.8338 

DEPENDENT  VARIABLE;  PM 


SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

27 

49.37805217 

1.82881675 

ERROR 

53 

46.62194783 

0.87965939 

CORRECTED  TOTAL 

80 

96.00000000 

MODEL  F  = 

2.08 

PR  >  F  «  0.0114 

R-SQUARE 

C.V. 

ROOT  MSE 

UM  MEAN 

0.514355 

211.0279 

0.93790159 

0.44444444 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

19.93888889 

2.83 

0.0108 

LOAD 

2 

6.80759573 

3.87 

0.0270 

DELAY 

3 

2.80484371 

1.06 

0.3727 

SECTOR 

2 

4.67829132 

2.66 

0.0793 

DELAY*SECTOR 

6 

1.60433921 

0.30 

0.9321 

LOAD* DELAY 

6 

13.54409331 

2.57 

0.0295 

SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

14.93757598 

2.12 

0.0496 

LOAD 

2 

6.96947066 

3.96 

0.0249 

DELAY 

3 

3.32085860 

1.26 

0.2981 

SECTOR 

2 

5.18954075 

2.95 

0.0610 

DELAY  *SECTOR 

6 

1.60433921 

0.30 

0.9321 

LOAD*DELAY 

6 

13.54409331 

2.57 

0.0295 
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DEPENDENT  VARIABLE:  S0+S1+S2 


SOURCE 

DP 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

27 

564.89653837 

20.92209401 

ERROR 

53 

190.83185669 

3.60060107 

CORRECTED  TOTAL 

80 

755.72839506 

MODEL  F  = 

5.81 

PR  >  F  =  0.0001 

R-SQUARE 

C.V. 

ROOT  MSE 

S0+S1+S2  MEAN 

0.747486 

55.4872 

1.89752499 

3.41975309 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

67.01728395 

2.33 

0.0321 

LOAD 

2 

209.95204018 

29.16 

0.0001 

DELAY 

3 

40.41697061 

3.74 

0.0164 

SECTOR 

2 

164.42427638 

22.83 

0.0001 

DELAY* SECTOR 

6 

22.28764182 

1.03 

0.4153 

LOAD* DELAY 

6 

60.79832545 

2.81 

0.0188 

SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

68.46576236 

2.38 

0.0288 

LOAD 

2 

227.33850000 

31.57 

0.0001 

DELAY 

3 

36.16478473 

3.35 

0.0258 

SECTOR 

2 

170.10625031 

23.62 

0.0001 

DELAY ‘SECTOR 

6 

22.28764182 

1.03 

0.4153 

LOAD* DELAY 

6 

60.79832545 

2.81 

0.0188 

LEAST 

SQUARES 

MEANS 

DELAY 

CM 

PROB  1 T 1 

HO:  LSMEAN (I) =LSMEAN(J) 

LSMEAN 

I/J  1 

2 

3 

4 

0 

2.94417300 

1 

• 

0.5803 

0.9295 

0.7913 

1 

3.28130655 

2 

0.5803 

• 

0.6645 

0.7929 

2 

3.00000000 

3 

0.9295 

0.6645 

• 

0.8675 

3 

3.11111111 

4 

0.7913 

0.7929 

0.8675 

• 
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DELAY 


0 

1 

2 

3 


UM 

LSMEAN 


PROB  | T |  HO:  LSMEAN (I) =LSMEAN(J) 
I/J  12  3  4 


0.26583424  1 
0.59971046  2 
0.77777778  3 
0.33333333  4 


0.2480 

0.2480 

0.0897  0.5608 

0.8206  0.3852 


0.0897  0.8206 

0.5608  0.3852 

0.1610 

0.1610 


DELAY 


0 

1 

2 

3 


S0+S1+S2 

LSMEAN 


PROB  | T |  HO:  LSMEAN ( I ) =LSMEAN ( J ) 
1/J  12  3  4 


3.14214622  1 
2.92987695  2 
4.72222222  3 
3.50000000  4 


0.7151 

0.7151 

0.0109  0.0052 

0.5528  0.3585 


0.0109  0.5528 

0.0052  0.3585 

0.0587 

0.0587 


NOTE:  TO  ENSURE  OVERALL  PROTECTION  LEVEL,  ONLY  PROBABILITIES 
ASSOCIATED  WITH  PRE-PLANNED  COMPARISONS  SHOULD  BE  USED. 


LOAD 

CM 

PROB  1 T 1 

HO:  LSMEAN (I) =LSMEAN(J) 

LSMEAN 

I/J  1 

2 

3 

1 

1.31944444 

1 

0.0012 

0.0001 

2 

3.20833333 

2  0.0012 

• 

0.0084 

3 

4.72466522 

3  0.0001 

0.0084 

• 

LOAD 

UM 

LSMEAN 

PROB  | T 1 
I/J  1 

HO:  LSMEAN (I) =LSMEAN(J) 
2  3 

1 

0.08333333 

1 

0.0468 

0.0093 

2 

0.61111111 

2  0.0468 

• 

0.5011 

3 

0.78804741 

3  0.0093 

0.5011 

• 
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LEAST  SQUARES  MEANS 


LOAD 

S0+S1+S2 

LSMEAN 

PROB  | T | 
I/J  1 

HO:  LSMEAN ( I) =LSMEAN(J) 
2  3 

1 

1.55555556 

1 

0.0008 

0.0001 

2 

3.41666667 

2  0.0008 

• 

0.0001 

3 

5.74846182 

3  0.0001 

0.0001 

• 

NOTES  TO  ENSURE  OVERALL  PROTECTION  LEVEL,  ONLY  PROBABILITIES 
ASSOCIATED  WITH  PRE-PLANNED  COMPARISONS  SHOULD  BE  USED. 


LOAD 

DELAY 

CM 

LSMEAN 

LSMEAN 

NUMBER 

1 

0 

0.77777778 

1 

1 

1 

1.88807456 

2 

1 

2 

1.25560984 

3 

1 

3 

1.35631560 

4 

2 

0 

3.00000000 

5 

2 

1 

2.68964893 

6 

2 

2 

3.38807456 

7 

2 

3 

3.75560984 

8 

3 

0 

5.05474122 

9 

3 

1 

5.26619616 

10 

3 

2 

4.35631560 

11 

3 

3 

4.22140789 

12 

PROB 

1  T  |  HO 

S  LSMEAN ( I ) =LSME AN f J ) 

I/J 

1 

2 

3 

4 

5 

6 

7 

8 

1 

• 

0.3011 

0.6547 

0.5881 

0.0214 

0.0775 

0.0174 

0.0071 

2 

0.3011 

• 

0.5971 

0.6562 

0.3004 

0.5027 

0.1970 

0.1223 

3 

0.6547 

0.5971 

• 

0.9326 

0.1065 

0.2317 

0.0787 

0.0339 

4 

0.5881 

0.6562 

0.9326 

• 

0.1276 

0.2506 

0.0930 

0.0480 

5 

0.0214 

0.3004 

0.1065 

0.1276 

• 

0.7712 

0.7166 

0.4800 

6 

0  0775 

0.5027 

0.2317 

0.2506 

0.7712 

• 

0.5591 

0.3725 

7 

0.0174 

0.1970 

0.0787 

0.0930 

0.7166 

0.5591 

• 

0.7585 

8 

0.0071 

0.1223 

0.0339 

0.0480 

0.4800 

0.3725 

0.7585 

• 

9 

0.0002 

0.0079 

0.0024 

0.0030 

0.0587 

0.0516 

0.1524 

0.2796 

10 

0.0001 

0.0029 

0.0004 

0.0009 

0.0216 

0.0237 

0.0884 

0.1572 

11 

0.0014 

0.0426 

0.0116 

0.0116 

0.2070 

0.1524 

0.4187 

0.6144 

12 

0.0021 

0.0471 

0.0158 

0.0194 

0.2558 

0.2028 

0.4711 

0.6969 
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PROB  | T |  HOi  LSME AN (I) =LSME AN (J) 


I/J 

9 

10 

11 

12 

1 

0.0002 

0.0001 

0.0014 

0.0021 

2 

0.0079 

0.0029 

0.0426 

0.0471 

3 

0.0024 

0.0004 

0.0116 

0.0158 

4 

0.0030 

0.0009 

0.0116 

0.0194 

5 

0.0587 

0.0216 

0.2070 

0.2558 

6 

0.0516 

0.0237 

0.1524 

0.2028 

7 

0.1524 

0.0884 

0.4187 

0.4711 

LEAST  SQUARES  MEANS  FOR  EFFECT  LOAD*DELAY 
PROB  | T |  HO:  LSMEAN (I)=LSMEAN (J) 


DEPENDENT  VARIABLE:  CM 


I/J 

9 

10 

11 

12 

8 

0.2796 

0.1572 

0.6144 

0.6969 

9 

• 

0.8458 

0.5591 

0.4711 

10 

0.8458 

• 

0.4144 

0.3386 

11 

0.5591 

0.4144 

• 

0.9100 

12 

0.4711 

0.3386 

0.9100 

• 

LEAST  SQUARES  MEANS 


LOAD  DELAY 


DM  LSMEAN 
LSMEAN  NUMBER 


1 

1 

1 

1 

2 

2 

2 

2 

3 

3 

3 

3 


0 

1 

2 

3 

0 

1 

2 

3 

0 

1 

2 

3 


-0.00000000  1 

0.18639:60  2 

0.02461093  3 

0.12233080  4 

0.44444444  5 

1.45566413  6 

0.35305827  7 

0.19127760  8 

0.35305827  9 

0.15707564  10 

1.95566413  11 

0.68639160  12 
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PROB  |Tj  HO:  LSME AN ( I ) =LSME AN IJ) 


I/J  1 

2 

3 

4 

5 

6 

7 

8 

1 

• 

0.7117 

0.9610 

0.8080 

0.3194 

0.0053 

0.4846 

0.7042 

2 

0.7117 

0.7742 

0.9094 

0.6091 

0.0276 

0.7594 

0.9931 

3 

0.9610 

0.7742 

• 

0.8619 

0.4059 

0.0134 

0.5607 

0.7594 

4 

0.8080 

0.9094 

0.8619 

• 

0.5229 

0.0171 

0.6822 

0.9023 

5 

0.3194 

0.6091 

0.4059 

0.5229 

0.0486 

0.8561 

0.6155 

6 

0.0053 

0.0276 

0.0134 

0.0171 

0.0486 

• 

0.0543 

0.0279 

7 

0.4846 

0.7594 

0.5607 

0.6822 

0.8561 

0.0543 

• 

0.7742 

8 

0.7042 

0.9931 

0.7594 

0.9023 

0.6155 

0.0279 

0.7742 

. 

9 

0.4846 

0.7594 

0.5607 

0.6822 

0.8561 

0.0543 

1.0000 

0.7742 

10 

0.7294 

0.9544 

0.7907 

0.9472 

0.5274 

0.0160 

0.7025 

0.9454 

11 

0.0003 

0.0026 

0.0011 

0.0013 

0.0039 

0.3600 

0.0060 

0.0026 

12 

0.1770 

0.3600 

0.2434 

0.3187 

0.6315 

0.1756 

0.5408 

0.3815 

PROB  1 T 1 

HO:  LSMEAN ( I ) =LSME AN ( J ) 

I/J 

9 

10 

11 

12 

1 

0.4846 

0.7294 

0.0003 

0.1770 

2 

0.7594 

0.9544 

0.0026 

0.3600 

3 

0.5607 

0.7907 

0.0011 

0.2434 

4 

0.6822 

0.9472 

0.0013 

0.3187 

5 

0.8561 

0.5274 

0.0039 

0.6315 

6 

0.0543 

0.0160 

0.3600 

0.1756 

7 

1.0000 

0.7025 

0.0060 

0.5408 

8 

0.7742 

0.9454 

0.0026 

0.3815 

9 

• 

0.7025 

0.0060 

0.5408 

10 

0.7025 

• 

0.0011 

0.3044 

11 

0.0060 

0.0011 

• 

0.0276 

12 

0.5408 

0.3044 

0.0276 

• 

LEAST 

SOUARES  MEANS 

LOAD 

DELAY 

S0+S1+S2 

LSMEAN 

LSMEAN 

NUMBER 

1 

0 

1.88888889 

1 

1 

1 

1.03754976 

2 

1 

2 

1.96045965 

3 

1 

3 

1.33532392 

4 

2 

0 

2 . 66666667 

5 

2 

1 

4 .33532392 

6 

2 

2 

3.87088310 

7 

2 

3 

2.79379298 

8 

3 

0 

4.87088310 

9 

3 

1 

3.41675715 

10 

3 

2 

8.33532392 

11 

3 

3 

6.37088310 

12 
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PROB  | T |  HO:  LSME AN ( I ) =LSMEAN ( J ) 


1/3 

1 

2 

3 

4 

5 

6 

7 

8 

1 

• 

0.4053 

0.9440 

0.5872 

0.3885 

0.0193 

0.0561 

0.3761 

2 

0.4053 

0.419b 

0.7938 

0.1144 

0.0053 

0.0125 

0.1277 

3 

0.9440 

0.4198 

# 

0.5828 

0.4891 

0.0405 

0.0982 

0.4502 

4 

0.5872 

0.7938 

0.5828 

• 

0.1946 

0.0084 

0.0295 

0.2028 

5 

0.3885 

0.1144 

0.4891 

0.1946 

0.1055 

0.2407 

0.9007 

6 

0.0193 

0.0053 

0.0405 

0.0084 

0.1055 

• 

0.6837 

0.1787 

7 

0.0561 

0.0125 

0.0982 

0.0295 

0.2407 

0.6837 

. 

0.3469 

8 

0.3761 

0.1277 

0.4502 

0.2028 

0.9007 

0.1787 

0.3469 

. 

9 

0.0049 

0.0010 

0.0132 

0.0029 

0.0343 

0.6386 

0.3655 

0.0729 

10 

0.1005 

0.0252 

0.1531 

0.0539 

0.4155 

0.3881 

0.6619 

0.5379 

11 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0006 

0.0002 

0.0001 

12 

0.0001 

0.0001 

0.0003 

0.0001 

0.0006 

0.0783 

0.0265 

0.0027 

PROB  | T | 

HO:  LSME AN ( I ) =LSME AN ( J ) 

I/J  9 

10 

11 

12 

1 

0.0049 

0.1005 

0.0001 

0.0001 

2 

0.0010 

0.0252 

0.0001 

0.0001 

3 

0.0132 

0.1531 

0.0001 

0.0003 

4 

0.0029 

0.0539 

0.0001 

0.0001 

5 

0.0343 

0.4155 

0.0001 

0.0006 

6 

0.6386 

0.3881 

0.0006 

0.0783 

7 

0.3655 

0.6619 

0.0002 

0.0265 

8 

0.0729 

0.5379 

0.0001 

0.0027 

9 

• 

0.1649 

0.0035 

0.1767 

10 

0.1649 

• 

0.0001 

0.0060 

11 

0.0035 

0.0001 

• 

0.0889 

12 

0.1767 

0.0060 

0.0889 

• 

NOTE:  TO  ENSURE  OVERALL  PROTECTION  LEVEL,  ONLY  PROBABILITIES 
ASSOCIATED  WITH  PRE-PLANNED  COMPARISONS  SHOULD  BE  USED. 


CLASS  LEVEL  INFORMATION 


CLASS 

LEVELS 

VALUES 

SUBJ 

9 

1 

2  3  4 

LOAD 

3 

1 

2  3 

DELAY 

4 

0 

12  3 

SECTOR 

3 

1 

2  3 

NUMBER  OF  OBSERVATIONS  IN  DATA  SET  =72 


F  -  9 


DEPENDENT  VARIABLE:  PROX 


SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

27 

25.54802184 

0.94622303 

ERROR 

44 

30.72975594 

0.69840354 

CORRECTED  TOTAL 

71 

56.27777778 

MODEL  F  = 

1.35 

PR  >  F  =  0.1817 

R-SQUARE 

O 

• 

< 

• 

ROOT  MSE 

PROX  MEAN 

0.453963 

150.4270 

0.83570542 

0.55555556 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

5.66666667 

1.01 

0.4395 

LOAD 

2 

8.64193098 

6.19 

0.0043 

DELAY 

3 

0.26962300 

0.13 

0.9426 

SECTOR 

2 

4.78169014 

3.42 

0.0415 

DELAY*SECTOR 

6 

3.66142626 

0.87 

0.5219 

LOAD* DELAY 

6 

2.52668479 

0.60 

0.7264 

SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

3.65278374 

0.65 

0.7284 

LOAD 

2 

6.68073801 

4.78 

0.0132 

DELAY 

3 

0.10189931 

0.05 

0.9856 

SECTOR 

2 

4.48813801 

3.21 

0.0498 

DELAY * SECTOR 

6 

3.66142626 

0.87 

0.5219 

LOAD* DELAY 

6 

2.52668479 

0.60 

0.7264 

F  - 10 


DELAY 


PROX  PROB  |  T  ]  HO:  LSMEAN  ( I )  =LSMEAN  ( J ) 
LSMEAN  I/J  12  3  4 


0  0.53161939  1  .  0.9749  0.9303  0.7668 

1  0.52245863  2  0.9749  .  0.9127  0.8036 

2  0.55555556  3  0.9303  0.9127  .  0.7155 

3  0.44355792  4  0.7668  0.8036  0.7155 


NOTE:  TO  ENSURE  OVERALL  PROTECTION  LEVEL,  ONLY  PROBABILITIES 
ASSOCIATED  WITH  PRE-PLANNED  COMPARISONS  SHOULD  BE  USED. 


PROB 


HO;  LSMEAN  (I)  g-LSMEAN  t  J 1 


I/J 

1 

2 

3 

4 

5 

6 

7 

8 

1 

• 

0.1598 

0.7121 

0.6806 

0.1167 

0.5738 

0.3211 

0.1392 

2 

0.1598 

• 

0.2956 

0.0885 

0.8055 

0.4409 

0.7123 

0.8831 

3 

0.7121 

0.2956 

• 

0.4480 

0.2248 

0.8241 

0.5249 

0.2481 

4 

0.6806 

0.0885 

0.4480 

. 

0.0560 

0.3405 

0.1926 

0.0740 

5 

0.1167 

0.8055 

0.2248 

0.0560 

• 

0.3233 

0.5575 

0.9238 

6 

0.5738 

0.4409 

0.8241 

0.3405 

0.3233 

• 

0.6939 

0.3768 

7 

0.3211 

0.7123 

0.5249 

0.1926 

0.5575 

0.6939 

• 

0.6252 

8 

0.1392 

0.8831 

0.2481 

0.0740 

0.9238 

0.3768 

0.6252 

9 

0.3467 

0.0265 

0.1996 

0.6191 

0.0196 

0.1623 

0.0703 

0.0235 

10 

0.6046 

0.4526 

0.8446 

0.3872 

0.3530 

0.9865 

0.6947 

0.3885 

11 

0.3752 

0.6953 

0.5833 

0.2198 

0.5507 

0.7497 

0.9589 

0.6074 

12 

0.4716 

0.5736 

0.7016 

0.2956 

0.4429 

0.8665 

0.8354 

0.5078 

F  - 1 1 


PROB  | T |  HO;  LSMEAN (I) =LSMEAN (J) 


I/J 

9 

10 

11 

12 

1 

0.3467 

0.6046 

0.3752 

0.4716 

2 

0.0265 

0.4526 

0.6953 

0.5736 

3 

0.1996 

0.8446 

0.5833 

0.7016 

4 

0.6191 

0.3872 

0.2198 

0.2956 

5 

0.0196 

0.3530 

0.5507 

0.4429 

6 

0.1623 

0.9865 

0.7497 

0.8665 

LEAST  SQUARES  MEANS  FOR  EFFECT  DELAY* SECTOR 
PROB  | T |  HO:  LSME AN ( I ) =LSME AN ( J ) 

DEPENDENT  VARIABLE:  PROX 


I/J 

9 

10 

11 

12 

7 

0.0703 

0.6947 

0.9589 

0.8354 

8 

0.0235 

0.3885 

0.6074 

0.5078 

9 

• 

0.1834 

0.0918 

0.1287 

10 

0.1834 

• 

0.7443 

0.8576 

11 

0.0918 

0.7443 

• 

0.8831 

12 

0.1287 

0.8576 

0.8831 

# 

NOTE:  TO  ENSURE  OVERALL  PROTECTION  LEVEL,  ONLY  PROBABILITIES 
ASSOCIATED  WITH  PRE-PLANNED  COMPARISONS  SHOULD  BE  USED 


LEAST 

SOUARES  MEANS 

LOAD 

DELAY 

PROX 

LSMEAN 

LSMEAN 

NUMBER 

1 

0 

0.52570922 

1 

1 

1 

-0.09751773 

2 

1 

2 

0.23847518 

3 

1 

3 

0.52304965 

4 

2 

0 

0.33333333 

5 

2 

1 

0.52304965 

6 

2 

2 

0.23581560 

7 

2 

3 

0.07180851 

8 

3 

0 

0.73581560 

9 

3 

1 

1.14184397 

10 

3 

2 

1.19237589 

11 

3 

3 

0.73581560 

12 

F- 12 


PROB 


|  T |  HO:  LSMEANm=LSMEAN(J) 


I/J  1 

2 

3 

4 

5 

6 

7 

8 

1 

. 

0.2221 

0.5696 

0.9965 

0.6692 

0.9965 

0.5675 

0.3702 

2 

0.2221 

« 

0.5109 

0.3118 

0.3428 

0.3118 

0.4933 

0.7399 

3 

0.5696 

0.5109 

• 

0.6605 

0.8335 

0.6605 

0.9958 

0.7314 

4 

0.9965 

0.3118 

0.6605 

• 

0.7462 

1.0000 

0.6381 

0.4869 

5 

0.6692 

0.3428 

0.8335 

0.7462 

0.7462 

0.8292 

0.5629 

6 

0.9965 

0.3118 

0.6605 

1.0000 

0.7462 

• 

0.6381 

0.4869 

7 

0.5675 

0.4933 

0.9958 

0.6381 

0.8292 

0.6381 

0.7478 

8 

0.3702 

0.7399 

0.7314 

0.4869 

0.5629 

0.4869 

0.7478 

9 

0.6783 

0.0912 

0.3319 

0.7274 

0.3752 

0.7274 

0.3057 

0.1970 

10 

0.1956 

0.0100 

0.0475 

0.3171 

0.0517 

0.3171 

0.0553 

0.0199 

11 

0.1740 

0.0139 

0.0637 

0.2793 

0.0612 

0.2793 

0.0639 

0.0305 

12 

0.6783 

0.0912 

0.3319 

0.7274 

0.3752 

0.7274 

0.3057 

0.1970 

PROB  | T |  HO:  LSME AN ( I ) =LSME AN ( J ) 


I/J  9 

10 

11 

12 

1 

0.6783 

0.1956 

0.1740 

0.6783 

2 

0.0912 

0.0100 

0.0139 

0.0912 

3 

0.3319 

0.0475 

0.0637 

0.3319 

4 

0.7274 

0.3171 

0.2793 

0.7274 

5 

0.3752 

0.0517 

0.0612 

0.3752 

6 

0.7274 

0.3171 

0.2793 

0.7274 

7 

0.3057 

0.0553 

0.0639 

0.3057 

8 

0.1970 

0.0199 

0.0305 

0.1970 

9 

• 

0.3824 

0.3692 

1.0000 

10 

0.3824 

• 

0.9147 

0.3824 

11 

0.3692 

0.9147 

. 

0.3692 

12 

1.0000 

0.3824 

0.3692 

• 

NOTE:  TO  ENSURE  OVERALL  PROTECTION  LEVEL,  ONLY  PROBABILITIES 
ASSOCIATED  WITH  PRE-PLANNED  COMPARISONS  SHOULD  BE  USED. 


CLASS  LEVEL  INFORMATION 


CLASS 

LEVELS 

VALUES 

SUBJ 

9 

1 

2  3  4 

LOAD 

3 

1 

2  3 

DELAY 

4 

0 

12  3 

SECTOR 

3 

1 

2  3 

NUMBER  OF  OBSERVATIONS  IN  DATA  SET  =81 


F-  13 


DEPENDENT  VARIABLE:  PP 


SOURCE 

DF 

SUM  OF  SQUARES 

KEAN  SQUARE 

MODEL 

27 

69.49846110 

2.57401708 

ERROR 

53 

95.85956359 

1.80867101 

CORRECTED  TOTAL 

80 

165.35802469 

MODEL  F  = 

1.42 

PR  >  F  =  0.1351 

R-SQUARE 

C.V. 

ROOT  MSE 

PP  MEAN 

0.420291 

96.4021 

1.34486840 

1.39506173 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

20.59135802 

1.42 

0.2087 

LOAD 

2 

22.62994350 

6.26 

0.0036 

DELAY 

3 

2.33348828 

0.43 

0.7323 

SECTOR 

2 

5.96601307 

1.65 

0.2019 

DELAY* SECTOR 

6 

11.44443781 

1.05 

0.4014 

LOAD* DELAY 

6 

6.53322040 

0.60 

0.7274 

SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

23.50154752 

1.62 

0.1401 

LOAD 

2 

22.38491911 

6.19 

0.0038 

DELAY 

3 

2.29705793 

0.42 

0.7370 

SECTOR 

2 

5.15200021 

1.42 

0.2497 

DELAY*SECTOR 

6 

11.44443781 

1.05 

0.4014 

LOAD*DELAY 

6 

6.53322040 

0.60 

0.7274 

DEPENDENT 

VARIABLE:  STEPS 

SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

27 

772.37823547 

28.60660131 

ERROR 

53 

382.75756700 

7.22184089 

CORRECTED 

TOTAL  80 

1155.13580247 

F  - 14 


MODEL  F 


3.96 


PR  >  F 


0.0001 


R-SQUARE 

C.V. 

ROOT  MSE 

STEPS  MEAN 

0.668647 

79.4435 

2.68734830 

3.38271605 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

75.91635802 

1.31 

0.2570 

LOAD 

2 

317.19285938 

21.96 

0.0001 

DELAY 

3 

138.33701638 

6.39 

0.0009 

SECTOR 

2 

64.79375973 

4.49 

0.0159 

DELAY  *SECTOR 

6 

75.23837046 

1.74 

0.1307 

LOAD* DELAY 

6 

100.89987149 

2.33 

0.0454 

SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

56.20671871 

0.97 

0.4672 

LOAD 

2 

334.47995495 

23.16 

0.0001 

DELAY 

3 

133.56629272 

6.16 

0.0011 

SECTOR 

2 

76.23502541 

5.28 

0.0081 

DELAY*SECTOR 

6 

75.23837046 

1.74 

0.1307 

LOAD* DELAY 

6 

100.89987149 

2.33 

0.0454 

DELAY 


0 

1 

2 

3 


LEAST  SQUARES  MEANS 


PP  PROB  | T |  HO:  LSMEAN ( I ) = LSMEAN ( J ) 
LSMEAN  I/J  12  3  4 


1.41250452  1 
1.47077452  2 
1.66666667  3 
1.16666667  4 


0.8875 

0.8875 

0.5520  0.6552 

0.5651  0.4888 


0.5520  0.5651 

0.6552  0.4888 

0.2697 

0.2697 


DELAY  STEPS  PROB  |T|  HO:  LSMEAN (I) =LSMEAN(J) 

LSMEAN  I/J  12  3  4 

0  2.18955845  1  .  0.8274  0.0020  0.0030 

1  2.36907347  2  0.8274  .  0.0047  0.0066 

2  4.94444444  3  0.0020  0.0047  .  0.9018 

3  4.83333333  4  0.0030  0.0066  0.9018 

NOTE:  TO  ENSURE  OVERALL  PROTECTION  LEVEL,  ONLY  PROBABILITIES 
ASSOCIATED  WITH  PRE-PLANNED  COMPARISONS  SHOULD  BE  USED. 


F- 15 


DELAY 

SECTOR 

PP 

LSMEAN 

LSMEAN 

NUMBER 

0 

1 

1.49101363 

1 

0 

2 

0.83745837 

2 

0 

3 

1.90904157 

3 

1 

1 

2.24218292 

4 

1 

2 

0.71472324 

5 

1 

3 

1.45541740 

6 

2 

1 

2.28491575 

7 

2 

2 

1.27403779 

8 

2 

3 

1.44104646 

9 

3 

1 

0.85648669 

10 

3 

2 

1.56695204 

11 

3 

3 

1.07656127 

12 

PROB 

o 

H 

:  LSMEAN1IJ  =LSMEAN  (J) 

I/J  1 

2 

3 

4 

5 

6 

7 

8 

1 

. 

0.3395 

0.5402 

0.2900 

0.2788 

0.9602 

0.2895 

0.7712 

2 

0.3395 

• 

0.1200 

0.0529 

0.8620 

0.3876 

0.0541 

0.5588 

3 

0.5402 

0.1200 

• 

0.6406 

0.0981 

0.5214 

0.6147 

0.3914 

4 

0.2900 

0.0529 

0.6406 

• 

0.0419 

0.2877 

0.9559 

0.2133 

5 

0.2788 

0.8620 

0.0981 

0.0419 

• 

0.3165 

0.0419 

0.4699 

6 

0.9602 

0.3876 

0.5214 

0.2877 

0.3165 

• 

0.2854 

0.8106 

7 

0.2895 

0.0541 

0.6147 

0.9559 

0.0419 

0.2854 

• 

0.2122 

8 

0.7712 

0.5588 

0.3914 

0.2133 

0.4699 

0.8106 

0.2122 

• 

9 

0.9460 

0.4197 

0.5310 

0.2923 

0.3490 

0.9851 

0.2966 

0.8355 

10 

0.3964 

0.9796 

0.1579 

0.0770 

0.8544 

0.4300 

0.0802 

0.5961 

11 

0.9181 

0.3301 

0.6468 

0.3740 

0.2724 

0.8852 

0.3698 

0.7160 

12 

0.5789 

0.7462 

0.2670 

0.1353 

0.6329 

0.6240 

0.1288 

0.8045 

F  - 16 


LEAST  SQUARES  MEANS  FOR  EFFECT  DELAY* SECTOR 


PROB  |T|  HO:  LSMEAN (I) =LBME AN (J) 


DEPENDENT  VARIABLE:  PP 


I/J  9 

10 

11 

12 

1 

0.9460 

0.3964 

0.9181 

0.5789 

2 

0.4197 

0.S796 

0.3301 

0.7462 

3 

0.5310 

0.1579 

0.6468 

0.2670 

4 

0.2923 

0.0770 

0.3740 

0.1353 

5 

0.3490 

0.8544 

0.2724 

0.6329 

6 

0.9851 

0.4300 

0.8852 

0.6240 

7 

0.2966 

0.0802 

0.3698 

0.1288 

8 

0.8355 

0.5961 

0.7160 

0.8045 

9 

• 

0.4647 

0.8729 

0.6508 

10 

0.4647 

0.3788 

0.7844 

11 

0.8729 

0.3788 

# 

0.5427 

12 

0.6508 

0.7844 

0.5427 

LEAST  SQUARES  MEANS 

DELAY  SECTOR  STEPS  LSMEAN 

LSMEAN  NUMBER 


0 

0 

0 

1 

1 

1 

2 

2 

2 

3 

3 

3 


1 

2 

3 

1 

2 

3 

1 

2 

3 

1 

2 

3 


2.24316372  1 
1.91634687  2 
2.40916477  3 
2.27759270  4 
2.33917563  5 
2.49045208  6 
6.09007230  7 
3.34842935  8 
5.39483168  9 
6.47650096  10 
1.18489221  11 
6.83860683  12 


F  - 17 


PROS  | T |  HO:  LSMEAN (I) =LSMEAN (J1 


I/J 

1 

2 

3 

4 

5 

6 

7 

8 

1 

• 

0.8103 

0.9030 

0.9805 

0.9463 

0.8622 

0.0122 

0.4594 

2 

0.8103 

• 

0.7175 

0.7998 

0.7645 

0.6871 

0.0063 

0.3386 

3 

0.9030 

0.7175 

• 

0.9264 

0.9608 

0.9541 

0.0163 

0.5251 

4 

0.9805 

0.7998 

0.9264 

• 

0.9666 

0.8849 

0.0162 

0.4887 

5 

0.9463 

0.7645 

0.9608 

0.9666 

• 

0.9181 

0.0158 

0.5138 

6 

0.8622 

0.6871 

0.9541 

0.8849 

0.9181 

. 

0.0229 

0.5710 

7 

0.0122 

0.0063 

0.0163 

0.0162 

0.0158 

0.0229 

* 

0.0924 

8 

0.4594 

0.3386 

0.5251 

0.4887 

0.5138 

0.5710 

0.0924 

9 

0.0364 

0.0228 

0.0492 

0.0432 

0.0518 

0.0640 

0.6656 

0.2063 

10 

0.0061 

0.0033 

0.0077 

0.0085 

0.0094 

0.0106 

0.8101 

0.0508 

11 

0.4742 

0.6239 

0.4128 

0.4710 

0.4555 

0.3991 

0.0032 

0.1821 

12 

0.0031 

0.0015 

0.0043 

0.0045 

0.0042 

0.0065 

0.6344 

0.0321 

PROB  | T |  HO:  LSME AN ( I ) =LSME AN ( J ) 


I/J  9 

10 

11 

12 

1 

0.0364 

0.0061 

0.4742 

0.0031 

2 

0.0228 

0.0033 

0.6239 

0.0015 

3 

0.0492 

0.0077 

0.4128 

0.0043 

4 

0.0432 

0.0085 

0.4710 

0.0045 

5 

0.0518 

0.0094 

0.4555 

0.0042 

6 

0.0640 

0.0106 

0.3991 

0.0065 

7 

0.6656 

0.8101 

0.0032 

0.6344 

8 

0.2063 

0.0508 

0.1821 

0.0321 

9 

• 

0.4982 

0.0095 

0.3710 

10 

0.4982 

• 

0.0017 

0.8218 

11 

0.0095 

0.0017 

• 

0.0009 

12 

0.3710 

0.8218 

0.0009 

• 

NOTES  TO  ENSURE  OVERALL  PROTECTION  LEVEL,  ONLY  PROBABILITIES 
ASSOCIATED  WITH  PRE-PLANNED  COMPARISONS  SHOULD  BE  USED. 


LEAST 

SOUARES  MEANS 

LOAD 

DELAY 

PP 

LSMEAN 

LSMEAN 

NUMBER 

1 

0 

0.88888889 

1 

1 

1 

0.51529135 

2 

1 

2 

1.37305465 

3 

1 

3 

0.11165400 

4 

2 

0 

1.33333333 

5 

2 

1 

1.61165400 

6 

2 

2 

2.01529135 

7 

2 

3 

1.20638798 

8 

3 

0 

2.01529135 

9 

3 

1 

2.28537821 

10 

3 

2 

1.61165400 

11 

3 

3 

2.18195802 

12 

F  - 18 


PROB 


I T  |  HO:  LSMEAN  m=LSMEANfJ) 


I/J  1 

2 

3 

4 

5 

6 

7 

8 

1 

• 

0.6056 

0.5034 

0.2841 

0.4863 

0.3188 

0.1233 

0.6604 

2 

0.6056 

* 

0.2911 

0.6175 

0.2605 

0.1782 

0.0587 

0.3941 

3 

0.5034 

0.2911 

• 

0.1216 

0.9561 

0.7671 

0.4282 

0.8309 

4 

0.2841 

0.6175 

0.1216 

• 

0.0948 

0.0587 

0.0215 

0.1778 

5 

0.4863 

0.2605 

0.9561 

0.0948 

• 

0.6999 

0.3473 

0.8604 

6 

0.3188 

0.1782 

0.7671 

0.0587 

0.6999 

• 

0.6175 

0.6153 

7 

0.1233 

0.0587 

0.4282 

0.0215 

0.3473 

0.6175 

0.3192 

8 

0.6604 

0.3941 

0.8309 

0.1778 

0.8604 

0.6153 

0.3192 

• 

9 

0.1233 

0.0587 

0.4282 

0.0215 

0.3473 

0.6175 

1.0000 

0.3192 

10 

0.0357 

0.0190 

0.2057 

0.0053 

0.1475 

0.3719 

0.7136 

0.1357 

11 

0.3188 

0.1782 

0.7671 

0.0587 

0.6999 

1.0000 

0.6175 

0.6153 

12 

0.0779 

0.0364 

0.3192 

0.0128 

0.2433 

0.4810 

0.8309 

0.2306 

PROB  | T |  HO:  LSMEAN ( I ) =LSMEAN ( J ) 


I/J  9 

10 

11 

12 

1 

0.1233 

0.0357 

0.3188 

0.0779 

2 

0.0587 

0.0190 

0.1782 

0.0364 

3 

0.4282 

0.2057 

0.7671 

0.3192 

4 

0.0215 

0.0053 

0.0587 

0.0128 

5 

0.3473 

0.1475 

0.6999 

0.2433 

6 

0.6175 

0.3719 

1.0000 

0.4810 

7 

1.0000 

0.7136 

0.6175 

0.8309 

8 

0.3192 

0.1357 

0.6153 

0.2306 

9 

• 

0.7136 

0.6175 

0.8309 

10 

0.7136 

• 

0.3719 

0.8882 

11 

0.6175 

0.3719 

• 

0.4810 

12 

0.8309 

0.8882 

0.4810 

• 

LEAST 

SOUARES  MEANS 

LOAD 

DELAY 

STEPS 

LSMEAN 

LSMEAN 

NUMBER 

1 

0 

1.22222222 

1 

1 

1 

1.06867535 

2 

1 

2 

2.01764387 

3 

1 

3 

2.74701412 

4 

2 

0 

I.44444444 

5 

2 

1 

2.74701412 

6 

2 

2 

3.06867535 

7 

2 

3 

2.68431053 

8 

3 

0 

3.90200869 

9 

3 

1 

3.29153094 

10 

3 

2 

9.74701412 

11 

3 

3 

9.06867535 

12 
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PROB  | T |  HO:  LSMEAN ( I ) =LBMEAN ( J) 


I/J 

1 

2 

3 

4 

5 

6 

7 

8 

1 

a 

0.9153 

0.5819 

0.2928 

0.8614 

0.2928 

0.2045 

0.3132 

2 

0.9153 

0.5574 

0.3006 

0.7948 

0.3006 

0.2030 

0.3194 

3 

0.5819 

0.5574 

0.6507 

0.6913 

0.6507 

0.5160 

0.6692 

4 

0.2928 

0.3006 

0.6507 

• 

0.3682 

1.0000 

0.8420 

0.9689 

5 

0.8614 

0.7948 

0.6913 

0.3682 

• 

0.3682 

0.2635 

0.3917 

6 

0.2928 

0.3006 

0.6507 

1.0000 

0.3682 

• 

0.8420 

0.9689 

7 

0.2045 

0.2030 

0.5160 

0.8420 

0.2635 

0.8420 

• 

0.8119 

8 

0.3132 

0.3194 

0.6692 

0.9689 

0.3917 

0.9689 

0.8119 

• 

9 

0.0678 

0.0735 

0.2463 

0.4751 

0.0931 

0.4751 

0.5934 

0.4521 

10 

0.1158 

0.1345 

0.3747 

0.7171 

0.1594 

0.7171 

0.8795 

0.6713 

11 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

12 

0.0001 

0.0001 

0.0001 

0.0002 

0.0001 

0.0002 

0.0003 

0.0002 

PROS  [ T |  HO:  LSMEAN ( I ) =LSMEAN ( J) 


I/J 

9 

10 

11 

12 

1 

0.0678 

0.1158 

0.0001 

0.0001 

2 

0.0735 

0.1345 

0.0001 

0.0001 

3 

0.2463 

0.3747 

0.0001 

0.0001 

4 

0.4751 

0.7171 

0.0001 

0.0002 

5 

0.0931 

0.1594 

0.0001 

0.0001 

6 

0.4751 

0.7171 

0.0001 

0.0002 

7 

0.5934 

0.8795 

0.0001 

0.0003 

8 

0.4521 

0.6713 

0.0001 

0.0002 

9 

# 

0.6780 

0.0006 

0.0016 

10 

0.6780 

• 

0.0001 

0.0002 

11 

0.0006 

0.0001 

* 

0.6744 

12 

0.0016 

0.0002 

0.6744 

• 

NOTES  TO  ENSURE  OVERALL  PROTECTION  LEVEL,  ONLY  PROBABILITIES 
ASSOCIATED  WITH  PRE-PLANNED  COMPARISONS  SHOULD  BE  USED. 


CLASS  LEVEL  INFORMATION 


CLASS 

LEVELS 

VALUES 

SUBJ 

9 

1  2  3  4  5 

6  7  8 

9 

DEL_LOAD 

12 

D0L1  D0L2 
D3L2  D3L3 

D0L3 

DILI 

D1L2 

D1L3 

D2L1 

D2L2 

D2L3 

D3L1 

DEL_SECT 

12 

D0S1  D0S2 
D3S2  D3S3 

D0S3 

D1S1 

D1S2 

D1S3 

D2S1 

D2S2 

D2S3 

D3S1 

NUMBER  OF  OBSERVATIONS  IN  DATA  SET  =81 


F  -  20 


DEPENDENT  VARIABLE:  STEPS 


SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

27 

772.37823547 

28.60660131 

ERROR 

53 

382.75756700 

7.22184089 

CORRECTED  TOTAL 

80 

1155.13580247 

MODEL  F  = 

3.96 

PR  >  F  =  0.0001 

R- SQUARE 

C.V. 

ROOT  MSE 

STEPS  MEAN 

0.668647 

79.4435 

2.68734830 

3.38271605 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

75.91635802 

1.31 

0.2570 

DEL  LOAD 

11 

556.42974726 

7.00 

0.0001 

DEL  SECT 

8 

140.03213019 

2.42 

0.0260 

SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

56.20671871 

0.97 

0.4672 

DEL  LOAD 

8 

410.00289540 

7.10 

0.0001 

DEL  SECT 

8 

140.03213019 

2.42 

0.0260 

F  -  21 


TUKEY 1 S  STUDENTIZED  RANGE  (HSD)  TEST  FOR  VARIABLE;  STEPS 


NOTES  THIS  TEST  CONTROLS  THE  TYPE  I  EXPERIMENTWISE  ERROR  RATE 


ALPHA=0 . 05  CONFIDENCE=0 . 95  DF=53  MSE=7. 22184 

CRITICAL  VALUE  OF  STUDENTIZED  RANGE=4.834 


COMPARISONS  SIGNIFICANT  AT  THE  0.05  LEVEL  ARE  INDICATED  BY  '***' 


SIMULTANEOUS 

SIMULTANEOUS 

LOWER 

DIFFERENCE 

UPPER 

DEL 

LOAD 

CONFIDENCE 

BETWEEN 

CONFIDENCE 

COMPARISON 

LIMIT 

MEANS 

LIMIT 

D2L3 

— 

D3L3 

-4.303 

1.000 

6.303 

D2L3 

- 

D0L3 

0.864 

6.167 

11.470 

*  *  * 

D2L3 

- 

D2L2 

1.697 

7.000 

12.303 

*  *  * 

D2L3 

- 

D3L1 

1.697 

7.000 

12.303 

*** 

D2L3 

- 

D1L3 

2.159 

7.000 

11.841 

*  *  * 

D2L3 

- 

D1L2 

1.697 

7.000 

12.303 

*  *  * 

D2L3 

- 

D3L2 

2.197 

7.500 

12.803 

*  *  * 

D2L3 

- 

D2L1 

2.864 

8.167 

13.470 

*** 

D2L3 

- 

D0L2 

3.715 

8.556 

13.397 

*** 

D2L3 

- 

D0L1 

3.937 

8.778 

13.619 

*  *  * 

D2L3 

— 

DILI 

3.697 

9.000 

14.303 

*** 

D3L3 

— 

D2L3 

-6.303 

-1.000 

4.303 

D3L3 

- 

D0L3 

-0.136 

5.167 

10.470 

D3L3 

- 

D2L2 

0.697 

6.000 

11.303 

*** 

D3L3 

- 

D3L1 

0.697 

6.000 

11.303 

*** 

D3L3 

- 

D1L3 

1.159 

6.000 

10.841 

*** 

D3L3 

- 

D1L2 

0.697 

6.000 

11.303 

*** 

D3L3 

- 

D3L2 

1.197 

6.500 

11.803 

*** 

D3L3 

- 

D2L1 

1.864 

7.167 

12.470 

*** 

D3L3 

- 

D0L2 

2.715 

7.556 

12 . 397 

*  *  * 

D3L3 

- 

D0L1 

2.937 

7.778 

12.619 

*** 

D3L3 

— 

DILI 

2.697 

8.000 

13.303 

*  *  * 

D0L3 

— 

D2L3 

-11.470 

-6.167 

-0.864 

*** 

D0L3 

- 

D3L3 

-10.470 

-5.167 

0.136 

D0L3 

- 

D2L2 

-4.470 

0.833 

6.136 

D0L3 

- 

D3L1 

-4.470 

0.833 

6.136 

D0L3 

- 

D1L3 

-4.008 

0.833 

5.674 

D0L3 

- 

D1L2 

-4.470 

0.833 

6.136 

D0L3 

- 

D3L2 

-3.970 

1.333 

6.636 

D0L3 

- 

D2L1 

-3.303 

2.000 

7.303 

D0L3 

- 

D0L2 

-2.452 

2.389 

7.230 

D0L3 

- 

D0L1 

-2.230 

2.611 

7.452 

D0L3 

- 

DILI 

-2.470 

2.833 

8.136 
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D2L2  -  D2L3 

-12.303 

-7.000 

-1.697 

D2L2  -  D3L3 

-11.303 

-6.000 

-0.697 

D2L2  -  D0L3 

-6.136 

-0.833 

4.470 

D2L2  -  D3L1 

-5.303 

0.000 

5.303 

D2L2  -  D1L3 

-4.841 

0.000 

4.841 

D2L2  -  D1L2 

-5.303 

0.000 

5.303 

DEL_LOAD 

COMPARISON 

D2L2  -  D3L2 
D2L2  -  D2L1 
D2L2  -  D0L2 
D2L2  -  D0L1 
D2L2  -  DILI 

D3L1  -  D2L3 
D3L1  -  D3L3 
D3L1  -  D0L3 
D3L1  -  D2L2 
D3LI  -  D1L3 
D3L1  -  D1L2 
D3L1  -  D3L2 
D3L1  -  D2L1 
D3L1  -  D0L2 
D3L1  -  D0L1 
D3L1  -  DILI 

D1L3  -  D2L3 
D1L3  -  D3L3 
D1L3  -  DOL3 
D1L3  -  D2L2 
D1L3  -  D3L1 
D1L3  -  D1L2 
D1L3  -  D3L2 
D1L3  -  D2L1 
D1L3  -  DOL2 
D1L3  -  DOL1 
D1L3  -  DILI 


SIMULTANEOUS 

LOWER 

CONFIDENCE 

LIMIT 

-4.803 

-4.136 

-3.285 

-3.063 

-3.303 

-12.303 

-11.303 

-6.136 

-5.303 

-4.841 

-5.303 

-4.803 

-4.136 

-3.285 

-3.063 

-3.303 

-11.841 

-10.841 

-5.674 

-4.841 

-4.841 

-4.841 

-4.341 

-3.674 

-2.774 

-2.552 

-2.841 


DIFFERENCE 

BETWEEN 

MEANS 

0.500 

1.167 

1.556 

1.778 

2.000 

-7.000 

-6.000 

-0.833 

0.000 

0.000 

0.000 

0.500 

1.167 

1.556 

1.778 

2.000 

-7.000 

-6.000 

-0.833 

0.000 

0.000 

0.000 

0.500 

1.167 

1.556 

1.778 

2.000 


SIMULTANEOUS 

UPPER 

CONFIDENCE 

LIMIT 

5.803 

6.470 
6.397 
6.619 

7.303 

-1.697 

-0.697 

4.470 

5.303 

4.841 

5.303 
5.803 

6.470 
6.397 
6.619 

7.303 

-2.159 

-1.159 

4.008 

4.841 

4.841 

4.841 
5.341 
6.008 
5.885 
6.108 

6.841 


*** 

*** 


*** 

*** 
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D1L2 

- 

D2L3 

-12.303 

-7.000 

-1.697 

D1L2 

- 

D3L3 

-11.303 

-6.000 

-0.697 

D1L2 

— 

D0L3 

-6.136 

-0.833 

4.470 

D1L2 

— 

D2L2 

-5.303 

0.000 

5.303 

D1L2 

- 

D3L1 

-5.303 

0.000 

5.303 

D1L2 

— 

D1L3 

-4.841 

0.000 

4.841 

D1L2 

— 

D3L2 

-4.803 

0.500 

5.803 

D1L2 

- 

D2L1 

-4.136 

1.167 

6.470 

D1L2 

D0L2 

-3.285 

1.556 

6.397 

D1L2 

— 

D0L1 

-3.063 

1.778 

6.619 

D1L2 

DILI 

-3.303 

2.000 

7.303 

D3L2 

D2L3 

-12.803 

-7.500 

-2.197 

D3L2 

— 

D3L3 

-11.803 

-6.500 

-1.197 

D3L2 

— 

DOL3 

-6.636 

-1.333 

3.970 

D3L2 

D2L2 

-5.803 

-0.500 

4.803 

D3L2 

— 

D3L1 

-5.803 

-0.500 

4.803 

D3L2 

— 

D1L3 

-5.341 

-0.500 

4.341 

D3L2 

— 

D1L2 

-5.803 

-0.500 

4.803 

D3L2 

— 

D2L1 

-4.636 

0.667 

5.970 

D3L2 

— 

DOL2 

-3.785 

1.056 

5.897 

SIMULTANEOUS 

SIMULTANEOUS 

LOWER 

DIFFERENCE 

UPPER 

DEL. 

LOAD 

CONFIDENCE 

BETWEEN 

CONFIDENCE 

COMPARISON 

LIMIT 

MEANS 

LIMIT 

D3L2 

- 

D0L1 

-3.563 

1.278 

6.119 

D3L2 

DILI 

-3.803 

1.500 

6.803 

D2L1 

D2L3 

-13.470 

-8.167 

-2.864 

D2L1 

— 

D3L3 

-12.470 

-7.167 

-1.864 

D2L1 

— 

DOL3 

-7.303 

-2.000 

3.303 

D2L1 

— 

D2L2 

-6.470 

-1.167 

4.136 

D2L1 

— 

D3L1 

-6.470 

-1.167 

4.136 

D2L1 

— 

D1L3 

-6.008 

-1.167 

3.674 

D2L1 

— 

D1L2 

-6.470 

-1.167 

4.136 

D2L1 

— 

D3L2 

-5.970 

-0.667 

4.636 

D2L1 

— 

D0L2 

-4.452 

0.389 

5.230 

D2L1 

— 

D0L1 

-4 .230 

0.611 

5.452 

D2L1 

— 

DILI 

-4.470 

0.833 

6.136 
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D0L2 

- 

D2L3 

-13.397 

-8.556 

-3.715 

D0L2 

- 

D3L3 

-12.397 

-7.556 

-2.715 

D0L2 

- 

DOL3 

-7.230 

-2 . 389 

2.452 

D0L2 

- 

D2L2 

-6.397 

-1.556 

3.285 

D0L2 

- 

D3L1 

-6.397 

-1.556 

3.285 

D0L2 

- 

D1L3 

-5.885 

-1.556 

2.774 

D0L2 

- 

D1L2 

-6.397 

-1.556 

3.285 

D0L2 

- 

D3L2 

-5.897 

-1.056 

3.785 

D0L2 

- 

D2L1 

-5.230 

-0.389 

4.452 

D0L2 

- 

DOL1 

-4.108 

0.222 

4 . 552 

D0L2 

— 

DILI 

-4.397 

0.444 

5.285 

D0L1 

— 

D2L3 

-13.619 

-8.778 

-3.937 

DOL1 

- 

D3L3 

-12.619 

-7.778 

-2.937 

DOL1 

- 

DOL3 

-7.452 

-2.611 

2.230 

DOL1 

- 

D2L2 

-6.619 

-1.778 

3.063 

DOL1 

- 

D3L1 

-6.619 

-1.778 

3 . 063 

DOL1 

- 

D1L3 

-6.108 

-1.778 

2.552 

DOL1 

- 

D1L2 

-6.619 

-1.778 

3.063 

DOL1 

- 

D3L2 

-6.119 

-1.278 

3.563 

DOL1 

- 

D2L1 

-5.452 

-0.611 

4.230 

DOL1 

- 

C0L2 

-4.552 

-0.222 

4.108 

DOL1 

- 

DILI 

-4.619 

0.222 

5.063 

DILI 

_ 

D2L3 

-14.303 

-9.000 

-3.697 

DILI 

- 

D3L3 

-13.303 

-8.000 

-2.697 

DILI 

- 

DOL3 

-8.136 

-2.833 

2.470 

DILI 

- 

D2L2 

-7.303 

-2.000 

3.303 

DILI 

- 

D3L1 

-7.303 

-2.000 

3.303 

DILI 

- 

D1L3 

-6.841 

-2 . 000 

2.841 

DILI 

- 

D1L2 

-7.303 

-2.000 

3.303 

DILI 

- 

D3L2 

-6.803 

-1.500 

3.803 

DILI 

- 

D2L1 

-6.136 

-0.833 

4.470 

DILI 

- 

D0L2 

-5.285 

-0.444 

4.397 

DILI 

- 

DOL1 

-5.063 

-0.222 

4.619 

*** 
★  ★  * 


★  1c  * 

★  *  ★ 


*** 
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TUKEY*  S  STUDENTIZED  RANGE  (HSD)  TEST  FOR  VARIABLE:  STEPS 


NOTE:  THIS  TEST  CONTROLS  THE  TYPE  I  EXPERIMENTWISE  ERROR  RATE 


ALPHA=0 .05  CONFIDENCE=0 . 95  DF=53  MSE=7. 22184 

CRITICAL  VALUE  OF  STUDENTIZED  RANGE=4.834 


COMPARISONS  SIGNIFICANT  AT  THE  0.05  LEVEL  ARE  INDICATED  BY  '***' 


SIMULTANEOUS  SIMULTANEOUS 


LOWER 

DIFFERENCE 

UPPER 

DEL 

SECT 

CONFIDENCE 

BETWEEN 

CONFIDEN< 

COMPARISON 

LIMIT 

MEANS 

LIMIT 

D3S3 

— 

D3S1 

-4.970 

0.333 

5.636 

D3S3 

- 

D2S1 

-4.636 

0.667 

5.970 

D3S3 

- 

D2S3 

-4.303 

1.000 

6.303 

D3S3 

- 

D2S2 

-1.803 

3.500 

8.803 

D3S3 

- 

D1S1 

-1.015 

4.095 

9.205 

D3S3 

- 

D1S3 

-0.872 

4.238 

9.348 

D3S3 

- 

D1S2 

-0.729 

4.381 

9.491 

D3S3 

- 

D0S1 

-0.419 

4.542 

9.502 

D3S3 

- 

D0S3 

-0.294 

4.667 

9.627 

D3S3 

- 

D0S2 

-0.044 

4.917 

9.877 

D3S3 

— 

D3S2 

-0.136 

5.167 

10.470 

D3S1 

— 

D3S3 

-5.636 

-0.333 

4.970 

D3S1 

- 

D2S1 

-4.970 

0.333 

5.636 

D3S1 

- 

D2S3 

-4.636 

0.667 

5.970 

D3S1 

- 

D2S2 

-2.136 

3.167 

8.470 

D3S1 

- 

D1S1 

-1.348 

3.762 

8.872 

D3S1 

- 

D1S3 

-1.205 

3.905 

9.015 

D3S1 

- 

D1S2 

-1.062 

4.048 

9.158 

D3S1 

- 

D0S1 

-0.752 

4.208 

9.169 

D3S1 

- 

D0S3 

-0.627 

4 . 333 

9.294 

D3S1 

- 

D0S2 

-0.377 

4.583 

9.544 

D3S1 

— 

D3S2 

-0.470 

4.833 

10.136 

D2S1 

— 

D3S3 

-5.970 

-0.667 

4.636 

D2S1 

- 

D3S1 

-5.636 

-0.333 

4.970 

D2S1 

- 

D2S3 

-4.970 

0.333 

5.636 

D2S1 

- 

D2S2 

-2.470 

2.833 

8.136 

D2S1 

- 

D1S1 

-1.682 

3.429 

8.539 

D2S1 

- 

D1S3 

-1.539 

3.571 

8.682 

D2S1 

- 

D1S2 

-1.396 

3.714 

8.824 

D2S1 

- 

D0S1 

-1.086 

3.875 

8 .836 

D2S1 

- 

D0S3 

-0.961 

4.000 

8.961 

D2S1 

- 

D0S2 

-0.711 

4.250 

9.211 

D2S1 

- 

D3S2 

-0.803 

4 . 500 

9.803 

F  -  26 


D2S3  -  D3S3 

-6.303 

-1.000 

4.303 

D2S3  -  D3S1 

-5.970 

-0.667 

4.636 

D2S3  -  D2S1 

-5.636 

-0.333 

4.970 

D2S3  -  D2S2 

-2.803 

2.500 

7.803 

D2S3  -  D1S1 

-2.015 

3.095 

8.205 

D2S3  -  D1S3 

-1.872 

3.238 

8.348 

SIMULTANEOUS 

SIMULTANEOUS 

LOWER 

DIFFERENCE 

UPPER 

DEL 

SECT 

CONFIDENCE 

BETWEEN 

CONFIDENCE 

COMPARISON 

LIMIT 

MEANS 

LIMIT 

D2S3 

— 

D1S2 

-1.729 

3 . 381 

8.491 

D2S3 

- 

D0S1 

-1.419 

3.542 

8.502 

D2S3 

- 

D0S3 

-1.294 

3.667 

8.627 

D2S3 

- 

D0S2 

-1.044 

3.917 

8.877 

D2S3 

— 

D3S2 

-1.136 

4.167 

9.470 

D2S2 

— 

D3S3 

-8.803 

-3.500 

1.803 

D2S2 

- 

D3S1 

-8.470 

-3.167 

2.136 

D2S2 

- 

D2S1 

-8.136 

-2.833 

2.470 

D2S2 

- 

D2S3 

-7.803 

-2.500 

2.803 

D2S2 

- 

D1S1 

-4.515 

0.595 

5.705 

D2S2 

- 

D1S3 

-4.372 

0.738 

5.848 

D2S2 

- 

D1S2 

-4.229 

0.881 

5.991 

D2S2 

- 

D0S1 

-3.919 

1.042 

6.002 

D2S2 

- 

DOS3 

-3.794 

1.167 

6.127 

D2S2 

- 

D0S2 

-3.544 

1.417 

6.377 

D2S2 

— 

D3S2 

-3.636 

1.667 

6.970 

D1S1 

— 

D3S3 

-9.205 

-4.095 

1.015 

D1S1 

- 

D3S1 

-8.872 

-3.762 

1.348 

D1S1 

- 

D2S1 

-8.539 

-3.429 

1.682 

D1S1 

- 

D2S3 

-8.205 

-3.095 

2.015 

D1S1 

- 

D2S2 

-5.705 

-0.595 

4.515 

D1S1 

- 

D1S3 

-4.767 

0.143 

5.052 

D1S1 

- 

D1S2 

-4.624 

0.286 

5.195 

D1S1 

- 

D0S1 

-4.307 

0.446 

5.200 

D1S1 

- 

D0S3 

-4.182 

0.571 

5.325 

D1S1 

- 

D0S2 

-3.932 

0.821 

5.575 

D1S1 

— 

D3S2 

-4.039 

1.071 

6.182 

D1S3 

— 

D3S3 

-9.348 

-4.238 

0.872 

D1S3 

- 

D3S1 

-9.015 

-3.905 

1.205 

D1S3 

- 

D2S1 

-8.682 

-3.571 

1.539 

D1S3 

- 

D2S3 

-8.348 

-3.238 

1.872 

D1S3 

- 

D2S2 

-5.848 

-0.738 

4.372 

D1S3 

- 

D1S1 

-5.052 

-0.143 

4.767 

D1S3 

- 

D1S2 

-4.767 

0.143 

5.052 

D1S3 

- 

D0S1 

-4.450 

0.304 

5.057 

D1S3 

- 

D0S3 

-4.325 

0.429 

5.182 

D1S3 

- 

D0S2 

-4.075 

0.679 

5.432 

D1S3 

- 

D3S2 

-4.182 

0.929 

6.039 

F  -  27 


D1S2 

- 

D3S3 

-9.491 

-4.381 

0.729 

D1S2 

- 

D3S1 

-9.158 

-4.048 

1.062 

D1S2 

- 

D2S1 

-8.824 

-3.714 

1.396 

D1S2 

- 

D2S3 

-8.491 

-3.381 

1.729 

D1S2 

- 

D2S2 

-5.991 

-0.881 

4.229 

D1S2 

- 

D1S1 

-5.195 

-0.286 

4.624 

D1S2 

- 

D1S3 

-5.052 

-0.143 

4.767 

D1S2 

- 

D0S1 

-4.593 

0.161 

4.914 

D1S2 

- 

DOS3 

-4.468 

0.286 

5.039 

SIMULTANEOUS 

SIMULTANEOUS 

LOWER 

DIFFERENCE 

UPPER 

DEL 

SECT 

CONFIDENCE 

BETWEEN 

CONFIDENCE 

COMPARISON 

LIMIT 

MEANS 

LIMIT 

D1S2 

— 

D0S2 

-4.218 

0.536 

5.289 

D1S2 

— 

D3S2 

-4.324 

0.786 

5.896 

D0S1 

— 

D3S3 

-9.502 

-4.542 

0.419 

D0S1 

- 

D3S1 

-9.169 

-4.208 

0.752 

D0S1 

- 

D2S1 

-8.836 

-3.875 

1.086 

D0S1 

- 

D2S3 

-8.502 

-3.542 

1.419 

D0S1 

- 

D2S2 

-6.002 

-1.042 

3.919 

D0S1 

- 

D1S1 

-5.200 

-0.446 

4.307 

D0S1 

- 

D1S3 

-5.057 

-0.304 

4.450 

D0S1 

- 

D1S2 

-4.914 

-0.161 

4.593 

D0S1 

- 

D0S3 

-4.468 

0.125 

4.718 

D0S1 

- 

D0S2 

-4.218 

0.375 

4.968 

D0S1 

— 

D3S2 

-4.336 

0.625 

5.586 

D0S3 

- 

D3S3 

-9.627 

-4.667 

0.294 

D0S3 

- 

D3S1 

-9.294 

-4.333 

0.627 

D0S3 

- 

D2S1 

-8.961 

-4.000 

0.961 

D0S3 

- 

D2S3 

-8.627 

-3.667 

1.294 

D0S3 

- 

D2S2 

-6.127 

-1.167 

3.794 

D0S3 

- 

D1S1 

-5.325 

-0.571 

4.182 

D0S3 

- 

D1S3 

-5.182 

-0.429 

4.325 

D0S3 

- 

D1S2 

-5.039 

-0.286 

4.468 

D0S3 

- 

D0S1 

-4.718 

-0.125 

4.468 

D0S3 

- 

D0S2 

-4.343 

0.250 

4.843 

D0S3 

— 

D3S2 

-4.461 

0.500 

5.461 

D0S2 

— 

D3S3 

-9.877 

-4.917 

0.044 

D0S2 

- 

D3S1 

-9.544 

-4.583 

0.377 

D0S2 

- 

D2S1 

-9.211 

-4.250 

0.711 

D0S2 

- 

D2S3 

-8.877 

-3.917 

1.044 

D0S2 

- 

D2S2 

-6.377 

-1.417 

3.544 

D0S2 

- 

D1S1 

-5.575 

-0.821 

3.932 

D0S2 

- 

D1S3 

-5.432 

-0.679 

4.075 

D0S2 

- 

D1S2 

-5.289 

-0.536 

4.218 

D0S2 

- 

D0S1 

-4.968 

-0.375 

4.218 

D0S2 

- 

D0S3 

-4.843 

-0.250 

4.343 

D0S2 

- 

D3S2 

-4.711 

0.250 

5.211 

F  -  28 


D3S2 

- 

D3S3 

-10.470 

-5.167 

0.136 

D3S2 

- 

D3S1 

-10.136 

-4.833 

0.470 

D3S2 

- 

D2S1 

-9.803 

-4.500 

0.803 

D3S2 

- 

D2S3 

-9.470 

-4.167 

1.136 

D3S2 

- 

D2S2 

-6.970 

-1.667 

3.636 

D3S2 

- 

D1S1 

-6.182 

-1.071 

4.039 

D3S2 

- 

D1S3 

-6.039 

-0.929 

4.182 

D3S2 

- 

D1S2 

-5.896 

-0.786 

4.324 

D3S2 

- 

D0S1 

-5.586 

-0.625 

4.336 

D3S2 

- 

D0S3 

-5.461 

-0.500 

4.461 

D3S2 

- 

D0S2 

-5.211 

-0.250 

4.711 

CLASS  LEVEL 

INFORMATION 

CLASS 

LEVELS 

VALUES 

SUBJ 

9 

1 

23456789 

SAT 

2 

0 

1 

LOAD 

2 

1 

2 

SECTOR 

3 

1 

2  3 

NUMBER  OF  OBSERVATIONS  IN  DATA  SET  =54 


DEPENDENT  VARIABLE:  STEPS 


SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

15 

91.94745990 

6.12983066 

ERROR 

38 

133.97846602 

3.52574911 

CORRECTED  TOTAL 

53 

225.92592593 

MODEL  F  = 

1.74 

PR  >  F  =  0.0842 

R-SQUARE 

C.V. 

ROOT  MSE 

STEPS  MEAN 

0.406981 

92.1779 

1.87769782 

2.03703704 

F  -  29 


SOURCE 

DP 

TYPE  I  SS 

P  VALUE 

PR  >  F 

SUBJ 

8 

46.92592593 

1.66 

0.1395 

SAT 

1 

15.80341880 

4.48 

0.0409 

LOAD 

1 

6.00000000 

1.70 

0.1999 

SECTOR 

2 

6.03703704 

0.86 

0.4328 

SAT* LOAD 

1 

0.32432432 

0.09 

0.7633 

SAT*SECTOR 

2 

16.85675381 

2.39 

0.1052 

SOURCE 

DP 

TYPE  III  SS 

P  VALUE 

PR  >  P 

SUBJ 

8 

50.12153398 

1.78 

0.1124 

SAT 

1 

15.80341880 

4.48 

0.0409 

LOAD 

1 

5.60432432 

1.59 

0.2151 

SECTOR 

2 

8.45417732 

1.20 

0.3127 

SAT* LOAD 

1 

0.32432432 

0.09 

0.7633 

SAT*SECTOR 

2 

16.85675381 

2.39 

0.1052 

LEAST  SQUARES  MEANS 


SAT  STEPS 

LSMEAN 

0  1.54700855 

1  2.64957265 


PROB  | T |  HO : 
LSMEAN1=LSMEAN2 

0.0409 


SAT  LOAD  STEPS  PROB  |T|  HO:  LSMEAN (I) =LSMEAN( J) 


LSMEAN 

I/J 

1 

2 

3 

4 

0 

1 

1.14160314 

1 

0.2526 

0.0984 

0.0218 

0 

2 

1.95241395 

2 

0.2526 

• 

0.5389 

0.2154 

1 

1 

2.40632941 

3 

0.0984 

0.5389 

0.5384 

1 

2 

2.89281589 

4 

0.0218 

0.2154 

0.5384 

NOTE:  TO  ENSURE  OVERALL  PROTECTION  LEVEL,  ONLY  PROBABILITIES 
ASSOCIATED  WITH  PRE-PLANNED  COMPARISONS  SHOULD  BE  USED 


GENERAL  LINEAR  MODELS  PROCEDURE 


DEPENDENT  VARIABLE:  Stress  reported  bv  Controllers 


SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

27 

67.17468250 

2. 

48795120 

ERROR 

53 

25.36852737 

0. 

47865146 

CORRECTED  TOTAL 

80 

92.54320988 

MODEL  F  = 

5.20 

PR  >  F 

=  0.0001 

R-SQUARE 

C.V. 

ROOT  MSE 

Stress  MEAN 

0.725874 

30.9611 

0.69184641 

2. 

23456790 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

13.72351291 

3.58 

0.0022 

DELAY 

3 

0.46760896 

0.33 

0.8068 

LOAD 

2 

35.24282608 

36.81 

0.0001 

SECTOR 

2 

13.40051754 

14.00 

0.0001 

DELAY* LOAD 

6 

2.65704994 

0.93 

0.4846 

DELAY*SECTOR 

6 

1.68316708 

0.59 

0.7398 

SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

13.38295393 

3.49 

0.0026 

DELAY 

3 

0.51458362 

0.36 

0.7833 

LOAD 

2 

35.62196614 

37.21 

0.0001 

SECTOR 

2 

13.59977015 

14.21 

0.0001 

DELAY* LOAD 

6 

2.55507194 

0.89 

0.5092 

DELAY  *SECTOR 

6 

1.68316708 

0.59 

0.7398 

SAS 

4 

11:54 

FRIDAY,  JUNE 

!  1,  1990 

F  -  31 


GENERAL  LINEAR  MODELS  PROCEDURE 


DEPENDENT  VARIABLE;  Attention  and  Concentration  (reported  bv  Controllers) 


SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

27 

54.74329907 

2.02752960 

ERROR 

53 

13.79991081 

0.26037568 

CORRECTED  TOTAL 

80 

68.54320988 

MODEL  F  = 

7.79 

PR  >  F  =  0.0001 

R- SQUARE 

C.V. 

ROOT  MSE 

Attn.  MEAN 

0.798668 

22.8353 

0.51027020 

2.23456790 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

6.60911897 

3.17 

0.0052 

DELAY 

3 

0.57011142 

0.73 

0.5388 

LOAD 

2 

34.14519835 

65.57 

0.0001 

SECTOR 

2 

10.02599836 

19.25 

0.0001 

DELAY* LOAD 

6 

0.79454041 

0.51 

0.7991 

DELAY* SECTOR 

6 

2.59833156 

1.66 

0.1484 

SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

8.86644329 

4.26 

0.0005 

DELAY 

3 

0.63981277 

0.82 

0.4891 

LOAD 

2 

34.05185060 

65.39 

0.0001 

SECTOR 

2 

10.08822210 

19.37 

0.0001 

DELAY* LOAD 

6 

0.82684016 

0.53 

0.7836 

DELAY*SECTOR 

6 

2.59833156 

1.66 

0.1484 

SAS 

5 

11:54 

FRIDAY,  JUNE 

1,  1990 

F  -  32 


GENERAL  LINEAR  MODELS  PROCEDURE 


LEAST 

SOUARES  MEANS 

DELAY 

Stress 

Atten. 

LSMEAN 

LSMEAN 

0 

2.18000218 

2.25679283 

1 

2.12326616 

2.15747694 

2 

2.33333333 

2 . 16666667 

3 

2.27777778 

2.38888889 

DEPENDENT 

GENERAL  : 

VARIABLE:  Stress 

LINEAR  MODELS  PROCEDURE 

(reported  bv  operations 

observers) 

SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

27 

47.59742740 

1.76286768 

ERROR 

53 

10.87170841 

0.20512657 

CORRECTED 

TOTAL  80 

58.46913580 

MODEL  F  = 

8.59 

PR  >  F  =  0.0001 

R-SQUARE 

C.V. 

ROOT  MSE 

Stress  MEAN 

0.814061 

26.3925 

0.45290901 

1.71604938 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

5.38580247 

3.28 

0.0041 

DELAY 

3 

2.98571213 

4.85 

0.0047 

LOAD 

2 

33.53021532 

81.73 

0.0001 

SECTOR 

2 

0.74449506 

1.81 

0.1729 

DELAY* LOAD 

6 

1.58264928 

1.29 

0.2797 

DELAY*SECTOR 

6 

3.36855314 

2.74 

0.0217 

F  -  33 


SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

3.65848046 

2.23 

0.0395 

DELAY 

3 

0.61357510 

1.00 

0.4014 

LOAD 

2 

33.29689903 

81.16 

0.0001 

SECTOR 

2 

1.01041205 

2.46 

0.0949 

DELAY* LOAD 

6 

1.59348765 

1.29 

0.2757 

DELAY*SECTOR 

6 

3.36855314 

2.74 

0.0217 

SAS 

4 

11:55 

FRIDAY,  JUNE 

1,  1990 

GENERAL 

LINEAR  MODELS  PROCEDURE 

DEPENDENT 

variable:  Attention  and 

Concentration 

(reported  by  operations 

observers) 

SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

MODEL 

27 

52.37633734 

1.93986435 

ERROR 

53 

10.51255155 

0.19835003 

CORRECTED 

TOTAL 

80 

62.88888889 

MODEL  F  = 

9.78 

PR  >  F  =  0.0001 

R-SQUARE 

C.V. 

ROOT  MSE 

Attn.  MEAN 

0.832839 

22.6884 

0.44536505 

1.96296296 

SOURCE 

DF 

TYPE  I  SS 

F  VALUE 

PR  >  F 

SUBJ 

8 

3.46944444 

2.19 

0.0433 

DELAY 

3 

0.68004366 

1.14 

0.3403 

LOAD 

2 

43.85681328 

110.55 

0.0001 

SECTOR 

2 

2.02358448 

5.10 

0.0094 

DELAY* LOAD 

6 

0.53280024 

0.45 

0.8434 

DELAY* SECTOR 

6 

1.81365123 

1.52 

0.1884 

F  -  34 


SOURCE 

DP 

TYPE  III  SS 

P  VALUE 

PR  >  P 

SUBJ 

8 

1.90085010 

1.20 

0.3182 

DELAY 

3 

0.37121704 

0.62 

0.6027 

LOAD 

2 

43.45867879 

109.55 

0.0001 

SECTOR 

2 

1.98381604 

5.00 

0.0102 

DELAY*LOAD 

6 

0.53222092 

0.45 

0.8437 

DELAY * SECTOR 

6 

1.81365123 

1.52 

0.1884 

SAS 

5 

11:55 

FRIDAY,  JUNE 

1,  1990 

GENERAL  LINEAR  MODELS  PROCEDURE 

LEAST 

SOUARES  MEANS 

DELAY 

Stress 

Atten. 

LSMEAN 

LSMEAN 

0 

1.63355438 

1.99100669 

1 

1.81613502 

1.89065118 

2 

1.61111111 

1.88888889 

3 

1.77777778 

2.05555556 

F-35/F-36 


Appendix  G 

Separation  Infringements  (Conflicts) 


Row# 

Run 

Week 

Day 

Sector 

#  Conflicts 

A 

B 

c 

i 

i 

i 

i 

i 

2 

1 

1 

2 

i 

i 

1 

2 

2 

0 

3 

3 

i 

i 

1 

3 

0 

1 

0 

4 

2 

i 

1 

1 

-- 

5 

2 

i 

i 

2 

- 

6 

2 

i 

i 

3 

— 

— 

— 

7 

3 

i 

i 

1 

- 

8 

3 

i 

i 

2 

.. 

9 

3 

i 

1 

3 

— 

10 

1 

i 

2 

1 

1 

0 

0 

11 

1 

i 

2 

2 

0 

0 

3 

12 

1 

- 

1 

2 

3 

0 

0 

0 

13 

2 

i 
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A  a  Conflict  between  two  aircraft  in  same  test  sector.  Sectors: 

B  =  Conflict  between  aircraft  in  different  test  sectors.  1  =  Dublin  High 

C  =  Conflict  with  one  aircraft  in  nontest  sector.  2  =  Sinca  Low 

Not  counted.  3  =  Macon  High 
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Run 

Week 

Day 

Sector 
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#  Conflicts 
ABC 


G  -  2 


A  U  S.  Government  Priming  Office:  1992-602-067 


G-3/G-4 


